2023-04-08 20:55:09,095 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/302c12cf-64ec-a05a-dc94-df5796107026 2023-04-08 20:55:09,111 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-08 20:55:09,127 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-08 20:55:09,127 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/302c12cf-64ec-a05a-dc94-df5796107026/cluster_f1ffd090-2f75-98d5-c8e5-9e16a1992906, deleteOnExit=true 2023-04-08 20:55:09,128 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-08 20:55:09,128 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/302c12cf-64ec-a05a-dc94-df5796107026/test.cache.data in system properties and HBase conf 2023-04-08 20:55:09,129 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/302c12cf-64ec-a05a-dc94-df5796107026/hadoop.tmp.dir in system properties and HBase conf 2023-04-08 20:55:09,129 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/302c12cf-64ec-a05a-dc94-df5796107026/hadoop.log.dir in system properties and HBase conf 2023-04-08 20:55:09,130 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/302c12cf-64ec-a05a-dc94-df5796107026/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-08 20:55:09,131 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/302c12cf-64ec-a05a-dc94-df5796107026/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-08 20:55:09,131 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-08 20:55:09,260 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-08 20:55:09,561 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-08 20:55:09,564 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/302c12cf-64ec-a05a-dc94-df5796107026/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-08 20:55:09,565 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/302c12cf-64ec-a05a-dc94-df5796107026/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-08 20:55:09,565 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/302c12cf-64ec-a05a-dc94-df5796107026/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-08 20:55:09,565 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/302c12cf-64ec-a05a-dc94-df5796107026/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-08 20:55:09,566 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/302c12cf-64ec-a05a-dc94-df5796107026/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-08 20:55:09,566 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/302c12cf-64ec-a05a-dc94-df5796107026/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-08 20:55:09,566 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/302c12cf-64ec-a05a-dc94-df5796107026/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-08 20:55:09,566 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/302c12cf-64ec-a05a-dc94-df5796107026/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-08 20:55:09,567 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/302c12cf-64ec-a05a-dc94-df5796107026/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-08 20:55:09,567 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/302c12cf-64ec-a05a-dc94-df5796107026/nfs.dump.dir in system properties and HBase conf 2023-04-08 20:55:09,567 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/302c12cf-64ec-a05a-dc94-df5796107026/java.io.tmpdir in system properties and HBase conf 2023-04-08 20:55:09,567 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/302c12cf-64ec-a05a-dc94-df5796107026/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-08 20:55:09,568 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/302c12cf-64ec-a05a-dc94-df5796107026/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-08 20:55:09,568 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/302c12cf-64ec-a05a-dc94-df5796107026/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-08 20:55:09,961 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-08 20:55:09,964 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-08 20:55:10,205 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-08 20:55:10,367 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-08 20:55:10,389 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-08 20:55:10,421 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-08 20:55:10,447 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/302c12cf-64ec-a05a-dc94-df5796107026/java.io.tmpdir/Jetty_localhost_localdomain_43193_hdfs____.p1w8ai/webapp 2023-04-08 20:55:10,561 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:43193 2023-04-08 20:55:10,589 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-08 20:55:10,590 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-08 20:55:10,935 WARN [Listener at localhost.localdomain/37581] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-08 20:55:11,024 WARN [Listener at localhost.localdomain/37581] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-08 20:55:11,038 WARN [Listener at localhost.localdomain/37581] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-08 20:55:11,043 INFO [Listener at localhost.localdomain/37581] log.Slf4jLog(67): jetty-6.1.26 2023-04-08 20:55:11,047 INFO [Listener at localhost.localdomain/37581] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/302c12cf-64ec-a05a-dc94-df5796107026/java.io.tmpdir/Jetty_localhost_39655_datanode____.1pe1gp/webapp 2023-04-08 20:55:11,122 INFO [Listener at localhost.localdomain/37581] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:39655 2023-04-08 20:55:11,402 WARN [Listener at localhost.localdomain/41823] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-08 20:55:11,751 DEBUG [Listener at localhost.localdomain/41823] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/302c12cf-64ec-a05a-dc94-df5796107026 2023-04-08 20:55:11,755 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x6dff351d4ef10f33: Processing first storage report for DS-1c4acb1f-0ba0-4fb9-baf5-61daf68e70fb from datanode 6ddeb571-9505-49d7-bdbc-e481d906fc0b 2023-04-08 20:55:11,756 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x6dff351d4ef10f33: from storage DS-1c4acb1f-0ba0-4fb9-baf5-61daf68e70fb node DatanodeRegistration(127.0.0.1:38891, datanodeUuid=6ddeb571-9505-49d7-bdbc-e481d906fc0b, infoPort=42279, infoSecurePort=0, ipcPort=41823, storageInfo=lv=-57;cid=testClusterID;nsid=772341962;c=1680987310037), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-04-08 20:55:11,756 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x6dff351d4ef10f33: Processing first storage report for DS-61d9b30b-79f0-406f-b1f9-0b01972fad25 from datanode 6ddeb571-9505-49d7-bdbc-e481d906fc0b 2023-04-08 20:55:11,757 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x6dff351d4ef10f33: from storage DS-61d9b30b-79f0-406f-b1f9-0b01972fad25 node DatanodeRegistration(127.0.0.1:38891, datanodeUuid=6ddeb571-9505-49d7-bdbc-e481d906fc0b, infoPort=42279, infoSecurePort=0, ipcPort=41823, storageInfo=lv=-57;cid=testClusterID;nsid=772341962;c=1680987310037), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-08 20:55:11,809 INFO [Listener at localhost.localdomain/41823] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/302c12cf-64ec-a05a-dc94-df5796107026/cluster_f1ffd090-2f75-98d5-c8e5-9e16a1992906/zookeeper_0, clientPort=52440, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/302c12cf-64ec-a05a-dc94-df5796107026/cluster_f1ffd090-2f75-98d5-c8e5-9e16a1992906/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/302c12cf-64ec-a05a-dc94-df5796107026/cluster_f1ffd090-2f75-98d5-c8e5-9e16a1992906/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-08 20:55:11,820 INFO [Listener at localhost.localdomain/41823] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=52440 2023-04-08 20:55:11,826 INFO [Listener at localhost.localdomain/41823] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-08 20:55:11,828 INFO [Listener at localhost.localdomain/41823] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-08 20:55:12,457 INFO [Listener at localhost.localdomain/41823] util.FSUtils(471): Created version file at hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c with version=8 2023-04-08 20:55:12,457 INFO [Listener at localhost.localdomain/41823] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/hbase-staging 2023-04-08 20:55:12,718 INFO [Listener at localhost.localdomain/41823] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-08 20:55:13,123 INFO [Listener at localhost.localdomain/41823] client.ConnectionUtils(127): master/jenkins-hbase20:0 server-side Connection retries=45 2023-04-08 20:55:13,153 INFO [Listener at localhost.localdomain/41823] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-08 20:55:13,154 INFO [Listener at localhost.localdomain/41823] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-08 20:55:13,154 INFO [Listener at localhost.localdomain/41823] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-08 20:55:13,154 INFO [Listener at localhost.localdomain/41823] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-08 20:55:13,155 INFO [Listener at localhost.localdomain/41823] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-08 20:55:13,268 INFO [Listener at localhost.localdomain/41823] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-08 20:55:13,333 DEBUG [Listener at localhost.localdomain/41823] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-08 20:55:13,408 INFO [Listener at localhost.localdomain/41823] ipc.NettyRpcServer(120): Bind to /148.251.75.209:46121 2023-04-08 20:55:13,417 INFO [Listener at localhost.localdomain/41823] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-08 20:55:13,420 INFO [Listener at localhost.localdomain/41823] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-08 20:55:13,436 INFO [Listener at localhost.localdomain/41823] zookeeper.RecoverableZooKeeper(93): Process identifier=master:46121 connecting to ZooKeeper ensemble=127.0.0.1:52440 2023-04-08 20:55:13,470 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:461210x0, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-08 20:55:13,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:46121-0x100925ce6e20000 connected 2023-04-08 20:55:13,497 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-08 20:55:13,497 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-08 20:55:13,502 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-08 20:55:13,512 DEBUG [Listener at localhost.localdomain/41823] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46121 2023-04-08 20:55:13,513 DEBUG [Listener at localhost.localdomain/41823] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46121 2023-04-08 20:55:13,513 DEBUG [Listener at localhost.localdomain/41823] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46121 2023-04-08 20:55:13,514 DEBUG [Listener at localhost.localdomain/41823] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46121 2023-04-08 20:55:13,515 DEBUG [Listener at localhost.localdomain/41823] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46121 2023-04-08 20:55:13,522 INFO [Listener at localhost.localdomain/41823] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c, hbase.cluster.distributed=false 2023-04-08 20:55:13,602 INFO [Listener at localhost.localdomain/41823] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-04-08 20:55:13,602 INFO [Listener at localhost.localdomain/41823] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-08 20:55:13,602 INFO [Listener at localhost.localdomain/41823] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-08 20:55:13,603 INFO [Listener at localhost.localdomain/41823] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-08 20:55:13,603 INFO [Listener at localhost.localdomain/41823] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-08 20:55:13,603 INFO [Listener at localhost.localdomain/41823] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-08 20:55:13,608 INFO [Listener at localhost.localdomain/41823] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-08 20:55:13,611 INFO [Listener at localhost.localdomain/41823] ipc.NettyRpcServer(120): Bind to /148.251.75.209:41091 2023-04-08 20:55:13,613 INFO [Listener at localhost.localdomain/41823] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-08 20:55:13,618 DEBUG [Listener at localhost.localdomain/41823] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-08 20:55:13,620 INFO [Listener at localhost.localdomain/41823] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-08 20:55:13,624 INFO [Listener at localhost.localdomain/41823] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-08 20:55:13,627 INFO [Listener at localhost.localdomain/41823] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:41091 connecting to ZooKeeper ensemble=127.0.0.1:52440 2023-04-08 20:55:13,641 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:410910x0, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-08 20:55:13,642 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ZKUtil(164): regionserver:410910x0, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-08 20:55:13,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:41091-0x100925ce6e20001 connected 2023-04-08 20:55:13,644 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-08 20:55:13,645 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-08 20:55:13,646 DEBUG [Listener at localhost.localdomain/41823] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=41091 2023-04-08 20:55:13,646 DEBUG [Listener at localhost.localdomain/41823] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=41091 2023-04-08 20:55:13,654 DEBUG [Listener at localhost.localdomain/41823] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=41091 2023-04-08 20:55:13,658 DEBUG [Listener at localhost.localdomain/41823] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=41091 2023-04-08 20:55:13,663 DEBUG [Listener at localhost.localdomain/41823] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=41091 2023-04-08 20:55:13,668 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase20.apache.org,46121,1680987312586 2023-04-08 20:55:13,677 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-08 20:55:13,678 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase20.apache.org,46121,1680987312586 2023-04-08 20:55:13,696 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-08 20:55:13,696 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-08 20:55:13,696 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-08 20:55:13,697 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-08 20:55:13,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-08 20:55:13,698 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase20.apache.org,46121,1680987312586 from backup master directory 2023-04-08 20:55:13,708 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase20.apache.org,46121,1680987312586 2023-04-08 20:55:13,708 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-08 20:55:13,709 WARN [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-08 20:55:13,709 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase20.apache.org,46121,1680987312586 2023-04-08 20:55:13,713 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-08 20:55:13,715 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-08 20:55:13,807 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/hbase.id with ID: 307296e4-c4f6-4c81-863b-64f9158ed679 2023-04-08 20:55:13,865 INFO [master/jenkins-hbase20:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-08 20:55:13,881 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-08 20:55:13,918 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x75b4962c to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:55:13,949 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@59beb6c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:55:13,967 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-08 20:55:13,968 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-08 20:55:13,983 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-08 20:55:13,983 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-08 20:55:13,985 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-08 20:55:13,988 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-08 20:55:13,989 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-08 20:55:14,020 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/MasterData/data/master/store-tmp 2023-04-08 20:55:14,045 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 20:55:14,045 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-08 20:55:14,045 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-08 20:55:14,045 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-08 20:55:14,045 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-08 20:55:14,045 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-08 20:55:14,046 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-08 20:55:14,046 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-08 20:55:14,047 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/MasterData/WALs/jenkins-hbase20.apache.org,46121,1680987312586 2023-04-08 20:55:14,068 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C46121%2C1680987312586, suffix=, logDir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/MasterData/WALs/jenkins-hbase20.apache.org,46121,1680987312586, archiveDir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/MasterData/oldWALs, maxLogs=10 2023-04-08 20:55:14,113 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38891,DS-1c4acb1f-0ba0-4fb9-baf5-61daf68e70fb,DISK] 2023-04-08 20:55:14,119 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 20:55:14,170 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/MasterData/WALs/jenkins-hbase20.apache.org,46121,1680987312586/jenkins-hbase20.apache.org%2C46121%2C1680987312586.1680987314077 2023-04-08 20:55:14,170 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38891,DS-1c4acb1f-0ba0-4fb9-baf5-61daf68e70fb,DISK]] 2023-04-08 20:55:14,171 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-08 20:55:14,171 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 20:55:14,173 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-08 20:55:14,174 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-08 20:55:14,222 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-08 20:55:14,228 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-08 20:55:14,247 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-08 20:55:14,258 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 20:55:14,264 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-08 20:55:14,265 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-08 20:55:14,283 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-08 20:55:14,294 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-08 20:55:14,296 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67339294, jitterRate=0.0034336745738983154}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-08 20:55:14,296 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-08 20:55:14,298 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-08 20:55:14,322 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-08 20:55:14,322 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-08 20:55:14,326 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-08 20:55:14,328 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-08 20:55:14,363 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 35 msec 2023-04-08 20:55:14,363 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-08 20:55:14,384 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-08 20:55:14,388 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-08 20:55:14,409 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-08 20:55:14,413 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-08 20:55:14,415 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-08 20:55:14,420 INFO [master/jenkins-hbase20:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-08 20:55:14,424 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-08 20:55:14,427 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-08 20:55:14,428 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-08 20:55:14,429 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-08 20:55:14,441 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-08 20:55:14,447 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-08 20:55:14,447 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-08 20:55:14,447 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-08 20:55:14,449 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase20.apache.org,46121,1680987312586, sessionid=0x100925ce6e20000, setting cluster-up flag (Was=false) 2023-04-08 20:55:14,464 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-08 20:55:14,468 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-08 20:55:14,469 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,46121,1680987312586 2023-04-08 20:55:14,475 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-08 20:55:14,479 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-08 20:55:14,480 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,46121,1680987312586 2023-04-08 20:55:14,482 WARN [master/jenkins-hbase20:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.hbase-snapshot/.tmp 2023-04-08 20:55:14,570 INFO [RS:0;jenkins-hbase20:41091] regionserver.HRegionServer(952): ClusterId : 307296e4-c4f6-4c81-863b-64f9158ed679 2023-04-08 20:55:14,574 DEBUG [RS:0;jenkins-hbase20:41091] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-08 20:55:14,579 DEBUG [RS:0;jenkins-hbase20:41091] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-08 20:55:14,579 DEBUG [RS:0;jenkins-hbase20:41091] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-08 20:55:14,582 DEBUG [RS:0;jenkins-hbase20:41091] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-08 20:55:14,583 DEBUG [RS:0;jenkins-hbase20:41091] zookeeper.ReadOnlyZKClient(139): Connect 0x5109cd44 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:55:14,586 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-08 20:55:14,592 DEBUG [RS:0;jenkins-hbase20:41091] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5fce3915, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:55:14,593 DEBUG [RS:0;jenkins-hbase20:41091] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5bf1a634, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-04-08 20:55:14,595 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-04-08 20:55:14,596 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-04-08 20:55:14,596 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-04-08 20:55:14,596 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-04-08 20:55:14,596 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase20:0, corePoolSize=10, maxPoolSize=10 2023-04-08 20:55:14,596 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-08 20:55:14,596 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-04-08 20:55:14,596 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-08 20:55:14,598 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1680987344598 2023-04-08 20:55:14,600 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-08 20:55:14,604 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-08 20:55:14,605 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-08 20:55:14,610 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-08 20:55:14,610 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-08 20:55:14,617 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-08 20:55:14,617 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-08 20:55:14,617 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-08 20:55:14,618 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-08 20:55:14,618 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-08 20:55:14,619 DEBUG [RS:0;jenkins-hbase20:41091] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase20:41091 2023-04-08 20:55:14,620 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-08 20:55:14,621 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-08 20:55:14,621 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-08 20:55:14,623 INFO [RS:0;jenkins-hbase20:41091] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-08 20:55:14,623 INFO [RS:0;jenkins-hbase20:41091] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-08 20:55:14,623 DEBUG [RS:0;jenkins-hbase20:41091] regionserver.HRegionServer(1023): About to register with Master. 2023-04-08 20:55:14,624 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-08 20:55:14,624 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-08 20:55:14,625 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1680987314625,5,FailOnTimeoutGroup] 2023-04-08 20:55:14,626 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1680987314626,5,FailOnTimeoutGroup] 2023-04-08 20:55:14,626 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-08 20:55:14,626 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-08 20:55:14,626 INFO [RS:0;jenkins-hbase20:41091] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase20.apache.org,46121,1680987312586 with isa=jenkins-hbase20.apache.org/148.251.75.209:41091, startcode=1680987313601 2023-04-08 20:55:14,627 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-08 20:55:14,628 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-08 20:55:14,643 DEBUG [RS:0;jenkins-hbase20:41091] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-08 20:55:14,683 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:36893, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-08 20:55:14,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:14,707 DEBUG [RS:0;jenkins-hbase20:41091] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c 2023-04-08 20:55:14,707 DEBUG [RS:0;jenkins-hbase20:41091] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost.localdomain:37581 2023-04-08 20:55:14,707 DEBUG [RS:0;jenkins-hbase20:41091] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-08 20:55:14,712 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-08 20:55:14,712 DEBUG [RS:0;jenkins-hbase20:41091] zookeeper.ZKUtil(162): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:14,712 WARN [RS:0;jenkins-hbase20:41091] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-08 20:55:14,713 INFO [RS:0;jenkins-hbase20:41091] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-08 20:55:14,714 DEBUG [RS:0;jenkins-hbase20:41091] regionserver.HRegionServer(1947): logDir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/WALs/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:14,715 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,41091,1680987313601] 2023-04-08 20:55:14,724 DEBUG [RS:0;jenkins-hbase20:41091] zookeeper.ZKUtil(162): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:14,734 DEBUG [RS:0;jenkins-hbase20:41091] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-08 20:55:14,743 INFO [RS:0;jenkins-hbase20:41091] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-08 20:55:14,762 INFO [RS:0;jenkins-hbase20:41091] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-08 20:55:14,765 INFO [RS:0;jenkins-hbase20:41091] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-08 20:55:14,766 INFO [RS:0;jenkins-hbase20:41091] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-08 20:55:14,766 INFO [RS:0;jenkins-hbase20:41091] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-08 20:55:14,772 INFO [RS:0;jenkins-hbase20:41091] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-08 20:55:14,773 DEBUG [RS:0;jenkins-hbase20:41091] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-08 20:55:14,773 DEBUG [RS:0;jenkins-hbase20:41091] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-08 20:55:14,773 DEBUG [RS:0;jenkins-hbase20:41091] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-08 20:55:14,773 DEBUG [RS:0;jenkins-hbase20:41091] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-08 20:55:14,774 DEBUG [RS:0;jenkins-hbase20:41091] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-08 20:55:14,774 DEBUG [RS:0;jenkins-hbase20:41091] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-04-08 20:55:14,774 DEBUG [RS:0;jenkins-hbase20:41091] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-08 20:55:14,774 DEBUG [RS:0;jenkins-hbase20:41091] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-08 20:55:14,774 DEBUG [RS:0;jenkins-hbase20:41091] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-08 20:55:14,774 DEBUG [RS:0;jenkins-hbase20:41091] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-08 20:55:14,776 INFO [RS:0;jenkins-hbase20:41091] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-08 20:55:14,776 INFO [RS:0;jenkins-hbase20:41091] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-08 20:55:14,776 INFO [RS:0;jenkins-hbase20:41091] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-08 20:55:14,791 INFO [RS:0;jenkins-hbase20:41091] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-08 20:55:14,793 INFO [RS:0;jenkins-hbase20:41091] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,41091,1680987313601-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-08 20:55:14,805 INFO [RS:0;jenkins-hbase20:41091] regionserver.Replication(203): jenkins-hbase20.apache.org,41091,1680987313601 started 2023-04-08 20:55:14,806 INFO [RS:0;jenkins-hbase20:41091] regionserver.HRegionServer(1638): Serving as jenkins-hbase20.apache.org,41091,1680987313601, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:41091, sessionid=0x100925ce6e20001 2023-04-08 20:55:14,806 DEBUG [RS:0;jenkins-hbase20:41091] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-08 20:55:14,806 DEBUG [RS:0;jenkins-hbase20:41091] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:14,806 DEBUG [RS:0;jenkins-hbase20:41091] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,41091,1680987313601' 2023-04-08 20:55:14,806 DEBUG [RS:0;jenkins-hbase20:41091] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:14,807 DEBUG [RS:0;jenkins-hbase20:41091] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:14,807 DEBUG [RS:0;jenkins-hbase20:41091] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-08 20:55:14,807 DEBUG [RS:0;jenkins-hbase20:41091] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-08 20:55:14,808 DEBUG [RS:0;jenkins-hbase20:41091] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:14,808 DEBUG [RS:0;jenkins-hbase20:41091] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,41091,1680987313601' 2023-04-08 20:55:14,808 DEBUG [RS:0;jenkins-hbase20:41091] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-08 20:55:14,808 DEBUG [RS:0;jenkins-hbase20:41091] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-08 20:55:14,808 DEBUG [RS:0;jenkins-hbase20:41091] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-08 20:55:14,809 INFO [RS:0;jenkins-hbase20:41091] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-08 20:55:14,809 INFO [RS:0;jenkins-hbase20:41091] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-08 20:55:14,923 INFO [RS:0;jenkins-hbase20:41091] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C41091%2C1680987313601, suffix=, logDir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/WALs/jenkins-hbase20.apache.org,41091,1680987313601, archiveDir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/oldWALs, maxLogs=32 2023-04-08 20:55:14,939 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38891,DS-1c4acb1f-0ba0-4fb9-baf5-61daf68e70fb,DISK] 2023-04-08 20:55:14,942 INFO [RS:0;jenkins-hbase20:41091] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/WALs/jenkins-hbase20.apache.org,41091,1680987313601/jenkins-hbase20.apache.org%2C41091%2C1680987313601.1680987314926 2023-04-08 20:55:14,943 DEBUG [RS:0;jenkins-hbase20:41091] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38891,DS-1c4acb1f-0ba0-4fb9-baf5-61daf68e70fb,DISK]] 2023-04-08 20:55:15,041 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-08 20:55:15,043 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-08 20:55:15,044 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c 2023-04-08 20:55:15,063 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 20:55:15,066 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-08 20:55:15,070 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/meta/1588230740/info 2023-04-08 20:55:15,070 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-08 20:55:15,071 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 20:55:15,072 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-08 20:55:15,074 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/meta/1588230740/rep_barrier 2023-04-08 20:55:15,074 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-08 20:55:15,075 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 20:55:15,075 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-08 20:55:15,077 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/meta/1588230740/table 2023-04-08 20:55:15,078 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-08 20:55:15,078 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 20:55:15,080 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/meta/1588230740 2023-04-08 20:55:15,081 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/meta/1588230740 2023-04-08 20:55:15,085 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-08 20:55:15,087 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-08 20:55:15,091 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-08 20:55:15,092 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65963061, jitterRate=-0.017073795199394226}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-08 20:55:15,092 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-08 20:55:15,092 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-08 20:55:15,093 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-08 20:55:15,093 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-08 20:55:15,093 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-08 20:55:15,093 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-08 20:55:15,094 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-08 20:55:15,094 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-08 20:55:15,100 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-08 20:55:15,100 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-08 20:55:15,106 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-08 20:55:15,116 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-08 20:55:15,119 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-08 20:55:15,271 DEBUG [jenkins-hbase20:46121] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-08 20:55:15,273 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,41091,1680987313601, state=OPENING 2023-04-08 20:55:15,279 DEBUG [PEWorker-5] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-08 20:55:15,280 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-08 20:55:15,281 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-08 20:55:15,284 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,41091,1680987313601}] 2023-04-08 20:55:15,470 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:15,472 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-08 20:55:15,477 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40872, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-08 20:55:15,488 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-08 20:55:15,489 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-08 20:55:15,493 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C41091%2C1680987313601.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/WALs/jenkins-hbase20.apache.org,41091,1680987313601, archiveDir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/oldWALs, maxLogs=32 2023-04-08 20:55:15,510 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38891,DS-1c4acb1f-0ba0-4fb9-baf5-61daf68e70fb,DISK] 2023-04-08 20:55:15,514 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/WALs/jenkins-hbase20.apache.org,41091,1680987313601/jenkins-hbase20.apache.org%2C41091%2C1680987313601.meta.1680987315495.meta 2023-04-08 20:55:15,515 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38891,DS-1c4acb1f-0ba0-4fb9-baf5-61daf68e70fb,DISK]] 2023-04-08 20:55:15,515 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-08 20:55:15,517 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-08 20:55:15,535 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-08 20:55:15,541 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-08 20:55:15,546 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-08 20:55:15,546 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 20:55:15,546 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-08 20:55:15,546 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-08 20:55:15,549 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-08 20:55:15,551 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/meta/1588230740/info 2023-04-08 20:55:15,551 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/meta/1588230740/info 2023-04-08 20:55:15,552 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-08 20:55:15,552 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 20:55:15,552 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-08 20:55:15,554 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/meta/1588230740/rep_barrier 2023-04-08 20:55:15,554 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/meta/1588230740/rep_barrier 2023-04-08 20:55:15,554 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-08 20:55:15,555 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 20:55:15,555 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-08 20:55:15,557 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/meta/1588230740/table 2023-04-08 20:55:15,557 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/meta/1588230740/table 2023-04-08 20:55:15,557 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-08 20:55:15,558 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 20:55:15,560 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/meta/1588230740 2023-04-08 20:55:15,563 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/meta/1588230740 2023-04-08 20:55:15,567 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-08 20:55:15,569 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-08 20:55:15,571 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67121231, jitterRate=1.8428266048431396E-4}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-08 20:55:15,571 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-08 20:55:15,581 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1680987315464 2023-04-08 20:55:15,596 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-08 20:55:15,597 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-08 20:55:15,598 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,41091,1680987313601, state=OPEN 2023-04-08 20:55:15,600 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-08 20:55:15,600 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-08 20:55:15,606 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-08 20:55:15,606 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,41091,1680987313601 in 316 msec 2023-04-08 20:55:15,611 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-08 20:55:15,611 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 501 msec 2023-04-08 20:55:15,616 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.0840 sec 2023-04-08 20:55:15,616 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1680987315616, completionTime=-1 2023-04-08 20:55:15,616 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-08 20:55:15,616 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-08 20:55:15,674 DEBUG [hconnection-0x3244987c-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:55:15,677 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40882, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:55:15,695 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-08 20:55:15,695 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1680987375695 2023-04-08 20:55:15,695 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1680987435695 2023-04-08 20:55:15,695 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 79 msec 2023-04-08 20:55:15,718 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,46121,1680987312586-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-08 20:55:15,719 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,46121,1680987312586-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-08 20:55:15,719 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,46121,1680987312586-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-08 20:55:15,720 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase20:46121, period=300000, unit=MILLISECONDS is enabled. 2023-04-08 20:55:15,721 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-08 20:55:15,726 DEBUG [master/jenkins-hbase20:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-08 20:55:15,735 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-08 20:55:15,736 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-08 20:55:15,745 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-08 20:55:15,748 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-08 20:55:15,751 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-08 20:55:15,772 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/hbase/namespace/301db7cfa588945a1b29ec427fecfa07 2023-04-08 20:55:15,774 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/hbase/namespace/301db7cfa588945a1b29ec427fecfa07 empty. 2023-04-08 20:55:15,775 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/hbase/namespace/301db7cfa588945a1b29ec427fecfa07 2023-04-08 20:55:15,775 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-08 20:55:16,199 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-08 20:55:16,203 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 301db7cfa588945a1b29ec427fecfa07, NAME => 'hbase:namespace,,1680987315735.301db7cfa588945a1b29ec427fecfa07.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp 2023-04-08 20:55:16,218 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680987315735.301db7cfa588945a1b29ec427fecfa07.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 20:55:16,218 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 301db7cfa588945a1b29ec427fecfa07, disabling compactions & flushes 2023-04-08 20:55:16,218 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680987315735.301db7cfa588945a1b29ec427fecfa07. 2023-04-08 20:55:16,218 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680987315735.301db7cfa588945a1b29ec427fecfa07. 2023-04-08 20:55:16,219 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680987315735.301db7cfa588945a1b29ec427fecfa07. after waiting 0 ms 2023-04-08 20:55:16,219 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680987315735.301db7cfa588945a1b29ec427fecfa07. 2023-04-08 20:55:16,219 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680987315735.301db7cfa588945a1b29ec427fecfa07. 2023-04-08 20:55:16,219 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 301db7cfa588945a1b29ec427fecfa07: 2023-04-08 20:55:16,227 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-08 20:55:16,240 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1680987315735.301db7cfa588945a1b29ec427fecfa07.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680987316230"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680987316230"}]},"ts":"1680987316230"} 2023-04-08 20:55:16,264 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-08 20:55:16,265 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-08 20:55:16,269 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680987316266"}]},"ts":"1680987316266"} 2023-04-08 20:55:16,276 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-08 20:55:16,283 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=301db7cfa588945a1b29ec427fecfa07, ASSIGN}] 2023-04-08 20:55:16,287 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=301db7cfa588945a1b29ec427fecfa07, ASSIGN 2023-04-08 20:55:16,289 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=301db7cfa588945a1b29ec427fecfa07, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,41091,1680987313601; forceNewPlan=false, retain=false 2023-04-08 20:55:16,441 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=301db7cfa588945a1b29ec427fecfa07, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:16,441 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1680987315735.301db7cfa588945a1b29ec427fecfa07.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680987316440"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680987316440"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680987316440"}]},"ts":"1680987316440"} 2023-04-08 20:55:16,447 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 301db7cfa588945a1b29ec427fecfa07, server=jenkins-hbase20.apache.org,41091,1680987313601}] 2023-04-08 20:55:16,615 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1680987315735.301db7cfa588945a1b29ec427fecfa07. 2023-04-08 20:55:16,617 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 301db7cfa588945a1b29ec427fecfa07, NAME => 'hbase:namespace,,1680987315735.301db7cfa588945a1b29ec427fecfa07.', STARTKEY => '', ENDKEY => ''} 2023-04-08 20:55:16,618 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 301db7cfa588945a1b29ec427fecfa07 2023-04-08 20:55:16,618 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680987315735.301db7cfa588945a1b29ec427fecfa07.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 20:55:16,618 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 301db7cfa588945a1b29ec427fecfa07 2023-04-08 20:55:16,619 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 301db7cfa588945a1b29ec427fecfa07 2023-04-08 20:55:16,621 INFO [StoreOpener-301db7cfa588945a1b29ec427fecfa07-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 301db7cfa588945a1b29ec427fecfa07 2023-04-08 20:55:16,624 DEBUG [StoreOpener-301db7cfa588945a1b29ec427fecfa07-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/namespace/301db7cfa588945a1b29ec427fecfa07/info 2023-04-08 20:55:16,624 DEBUG [StoreOpener-301db7cfa588945a1b29ec427fecfa07-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/namespace/301db7cfa588945a1b29ec427fecfa07/info 2023-04-08 20:55:16,624 INFO [StoreOpener-301db7cfa588945a1b29ec427fecfa07-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 301db7cfa588945a1b29ec427fecfa07 columnFamilyName info 2023-04-08 20:55:16,625 INFO [StoreOpener-301db7cfa588945a1b29ec427fecfa07-1] regionserver.HStore(310): Store=301db7cfa588945a1b29ec427fecfa07/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 20:55:16,627 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/namespace/301db7cfa588945a1b29ec427fecfa07 2023-04-08 20:55:16,628 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/namespace/301db7cfa588945a1b29ec427fecfa07 2023-04-08 20:55:16,633 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 301db7cfa588945a1b29ec427fecfa07 2023-04-08 20:55:16,637 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/namespace/301db7cfa588945a1b29ec427fecfa07/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-08 20:55:16,638 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 301db7cfa588945a1b29ec427fecfa07; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72169666, jitterRate=0.0754118263721466}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-08 20:55:16,638 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 301db7cfa588945a1b29ec427fecfa07: 2023-04-08 20:55:16,640 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1680987315735.301db7cfa588945a1b29ec427fecfa07., pid=6, masterSystemTime=1680987316602 2023-04-08 20:55:16,644 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1680987315735.301db7cfa588945a1b29ec427fecfa07. 2023-04-08 20:55:16,644 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1680987315735.301db7cfa588945a1b29ec427fecfa07. 2023-04-08 20:55:16,646 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=301db7cfa588945a1b29ec427fecfa07, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:16,646 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1680987315735.301db7cfa588945a1b29ec427fecfa07.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680987316645"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680987316645"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680987316645"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680987316645"}]},"ts":"1680987316645"} 2023-04-08 20:55:16,654 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-08 20:55:16,654 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 301db7cfa588945a1b29ec427fecfa07, server=jenkins-hbase20.apache.org,41091,1680987313601 in 203 msec 2023-04-08 20:55:16,658 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-08 20:55:16,658 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=301db7cfa588945a1b29ec427fecfa07, ASSIGN in 371 msec 2023-04-08 20:55:16,659 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-08 20:55:16,660 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680987316660"}]},"ts":"1680987316660"} 2023-04-08 20:55:16,663 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-08 20:55:16,666 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-08 20:55:16,668 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 929 msec 2023-04-08 20:55:16,750 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-08 20:55:16,751 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-08 20:55:16,752 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-08 20:55:16,790 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-08 20:55:16,807 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-08 20:55:16,812 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 30 msec 2023-04-08 20:55:16,824 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-08 20:55:16,836 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-08 20:55:16,841 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 17 msec 2023-04-08 20:55:16,853 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-08 20:55:16,855 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-08 20:55:16,855 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.145sec 2023-04-08 20:55:16,858 INFO [master/jenkins-hbase20:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-08 20:55:16,860 INFO [master/jenkins-hbase20:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-08 20:55:16,860 INFO [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-08 20:55:16,861 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,46121,1680987312586-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-08 20:55:16,862 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,46121,1680987312586-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-08 20:55:16,874 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-08 20:55:16,876 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x4c9bc423 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:55:16,880 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7854c900, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:55:16,892 DEBUG [hconnection-0xac2f484-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:55:16,904 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40898, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:55:16,913 INFO [Listener at localhost.localdomain/41823] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase20.apache.org,46121,1680987312586 2023-04-08 20:55:16,932 INFO [Listener at localhost.localdomain/41823] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=205, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=128, ProcessCount=175, AvailableMemoryMB=4019 2023-04-08 20:55:16,947 DEBUG [Listener at localhost.localdomain/41823] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-08 20:55:16,950 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:53136, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-08 20:55:16,959 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-08 20:55:16,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-08 20:55:16,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-08 20:55:16,966 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-08 20:55:16,969 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-08 20:55:16,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-08 20:55:16,972 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:16,974 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040 empty. 2023-04-08 20:55:16,975 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:16,975 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-08 20:55:16,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-08 20:55:17,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-08 20:55:17,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-08 20:55:17,394 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-08 20:55:17,398 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 7f864e4ec7cff383a8a9db2371cff040, NAME => 'TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp 2023-04-08 20:55:17,412 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 20:55:17,413 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 7f864e4ec7cff383a8a9db2371cff040, disabling compactions & flushes 2023-04-08 20:55:17,413 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:17,413 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:17,413 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. after waiting 0 ms 2023-04-08 20:55:17,413 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:17,413 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:17,413 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:17,418 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-08 20:55:17,421 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987317421"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680987317421"}]},"ts":"1680987317421"} 2023-04-08 20:55:17,425 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-08 20:55:17,427 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-08 20:55:17,428 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680987317427"}]},"ts":"1680987317427"} 2023-04-08 20:55:17,429 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-08 20:55:17,433 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7f864e4ec7cff383a8a9db2371cff040, ASSIGN}] 2023-04-08 20:55:17,436 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7f864e4ec7cff383a8a9db2371cff040, ASSIGN 2023-04-08 20:55:17,439 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=7f864e4ec7cff383a8a9db2371cff040, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,41091,1680987313601; forceNewPlan=false, retain=false 2023-04-08 20:55:17,591 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=7f864e4ec7cff383a8a9db2371cff040, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:17,592 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987317591"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680987317591"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680987317591"}]},"ts":"1680987317591"} 2023-04-08 20:55:17,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-08 20:55:17,600 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601}] 2023-04-08 20:55:17,765 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:17,766 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7f864e4ec7cff383a8a9db2371cff040, NAME => 'TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.', STARTKEY => '', ENDKEY => ''} 2023-04-08 20:55:17,766 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:17,766 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 20:55:17,767 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:17,767 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:17,769 INFO [StoreOpener-7f864e4ec7cff383a8a9db2371cff040-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:17,771 DEBUG [StoreOpener-7f864e4ec7cff383a8a9db2371cff040-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A 2023-04-08 20:55:17,772 DEBUG [StoreOpener-7f864e4ec7cff383a8a9db2371cff040-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A 2023-04-08 20:55:17,773 INFO [StoreOpener-7f864e4ec7cff383a8a9db2371cff040-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 20:55:17,773 INFO [StoreOpener-7f864e4ec7cff383a8a9db2371cff040-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7f864e4ec7cff383a8a9db2371cff040 columnFamilyName A 2023-04-08 20:55:17,774 INFO [StoreOpener-7f864e4ec7cff383a8a9db2371cff040-1] regionserver.HStore(310): Store=7f864e4ec7cff383a8a9db2371cff040/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 20:55:17,774 INFO [StoreOpener-7f864e4ec7cff383a8a9db2371cff040-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:17,776 DEBUG [StoreOpener-7f864e4ec7cff383a8a9db2371cff040-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B 2023-04-08 20:55:17,776 DEBUG [StoreOpener-7f864e4ec7cff383a8a9db2371cff040-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B 2023-04-08 20:55:17,777 INFO [StoreOpener-7f864e4ec7cff383a8a9db2371cff040-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 20:55:17,777 INFO [StoreOpener-7f864e4ec7cff383a8a9db2371cff040-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7f864e4ec7cff383a8a9db2371cff040 columnFamilyName B 2023-04-08 20:55:17,778 INFO [StoreOpener-7f864e4ec7cff383a8a9db2371cff040-1] regionserver.HStore(310): Store=7f864e4ec7cff383a8a9db2371cff040/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 20:55:17,778 INFO [StoreOpener-7f864e4ec7cff383a8a9db2371cff040-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:17,780 DEBUG [StoreOpener-7f864e4ec7cff383a8a9db2371cff040-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C 2023-04-08 20:55:17,780 DEBUG [StoreOpener-7f864e4ec7cff383a8a9db2371cff040-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C 2023-04-08 20:55:17,780 INFO [StoreOpener-7f864e4ec7cff383a8a9db2371cff040-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 20:55:17,781 INFO [StoreOpener-7f864e4ec7cff383a8a9db2371cff040-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7f864e4ec7cff383a8a9db2371cff040 columnFamilyName C 2023-04-08 20:55:17,781 INFO [StoreOpener-7f864e4ec7cff383a8a9db2371cff040-1] regionserver.HStore(310): Store=7f864e4ec7cff383a8a9db2371cff040/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 20:55:17,782 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:17,784 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:17,785 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:17,788 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-08 20:55:17,790 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:17,793 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-08 20:55:17,794 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 7f864e4ec7cff383a8a9db2371cff040; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62158776, jitterRate=-0.07376205921173096}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-08 20:55:17,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:17,795 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040., pid=11, masterSystemTime=1680987317754 2023-04-08 20:55:17,798 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:17,798 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:17,799 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=7f864e4ec7cff383a8a9db2371cff040, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:17,800 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987317799"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680987317799"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680987317799"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680987317799"}]},"ts":"1680987317799"} 2023-04-08 20:55:17,806 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-08 20:55:17,806 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 in 202 msec 2023-04-08 20:55:17,810 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-08 20:55:17,810 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=7f864e4ec7cff383a8a9db2371cff040, ASSIGN in 373 msec 2023-04-08 20:55:17,811 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-08 20:55:17,812 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680987317812"}]},"ts":"1680987317812"} 2023-04-08 20:55:17,814 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-08 20:55:17,817 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-08 20:55:17,819 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 855 msec 2023-04-08 20:55:18,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-08 20:55:18,104 INFO [Listener at localhost.localdomain/41823] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-08 20:55:18,114 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x1bf45e56 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:55:18,118 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@428c18bf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:55:18,120 DEBUG [Listener at localhost.localdomain/41823] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:55:18,124 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40910, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:55:18,127 DEBUG [Listener at localhost.localdomain/41823] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-08 20:55:18,130 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:53142, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-08 20:55:18,137 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x3b15648c to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:55:18,142 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@754370ad, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:55:18,142 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x2398b790 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:55:18,147 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5d114eb9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:55:18,148 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x1ee17908 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:55:18,151 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6194be61, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:55:18,152 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x554a753e to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:55:18,155 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4d5a66f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:55:18,156 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x0e18c51d to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:55:18,159 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7de00027, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:55:18,160 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x29fd8399 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:55:18,164 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@9048085, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:55:18,164 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x1232cedd to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:55:18,200 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@57966649, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:55:18,201 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x7746d7a7 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:55:18,209 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13065328, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:55:18,209 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x7124b8aa to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:55:18,217 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@309eb2c6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:55:18,226 DEBUG [hconnection-0x33489bfb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:55:18,226 DEBUG [hconnection-0x71ab225e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:55:18,227 DEBUG [hconnection-0x6b777934-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:55:18,230 DEBUG [hconnection-0x47478bd7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:55:18,231 DEBUG [hconnection-0x70178ded-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:55:18,231 DEBUG [hconnection-0x2a85a31e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:55:18,231 DEBUG [hconnection-0x29d6c97d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:55:18,231 DEBUG [hconnection-0x1a54083f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:55:18,233 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40930, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:55:18,233 DEBUG [hconnection-0x5c87bff5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:55:18,235 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40926, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:55:18,240 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40934, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:55:18,261 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40950, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:55:18,261 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40936, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:55:18,266 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40954, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:55:18,267 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40968, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:55:18,267 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40976, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:55:18,280 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40978, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:55:18,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:18,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:18,307 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:18,307 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:18,308 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:18,308 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:18,308 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:18,308 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:18,309 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:18,310 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:18,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:18,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:18,310 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:18,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:18,310 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:18,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:18,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:18,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:18,311 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:18,326 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:18,342 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:18,342 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:18,344 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:18,346 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:18,346 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:18,348 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:18,349 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:18,351 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7f864e4ec7cff383a8a9db2371cff040 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-08 20:55:18,363 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=A 2023-04-08 20:55:18,375 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:18,376 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=B 2023-04-08 20:55:18,376 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:18,376 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=C 2023-04-08 20:55:18,376 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:18,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:18,407 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:18,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680987318480, max timestamp=1680987318510Num uniques 859; 2023-04-08 20:55:18,550 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 20:55:18,556 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680987318480, max timestamp=1680987318510Num uniques 859; 2023-04-08 20:55:18,556 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-08 20:55:18,562 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680987318480, max timestamp=1680987318519Num uniques 859; 2023-04-08 20:55:18,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-08 20:55:18,609 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:18,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40950 deadline: 1680987378597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:18,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9975, totalHeapSize=22912, min timestamp=1680987318593, max timestamp=1680987318601Num uniques 1009; 2023-04-08 20:55:18,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46210, new segment data size=9975, suffix heap size=132904,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=218 2023-04-08 20:55:18,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680987318593, max timestamp=1680987318601Num uniques 1009; 2023-04-08 20:55:18,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=9978, suffix heap size=132904,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=218 2023-04-08 20:55:18,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680987318593, max timestamp=1680987318599Num uniques 1018; 2023-04-08 20:55:18,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-08 20:55:18,671 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:18,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40978 deadline: 1680987378663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:18,673 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:18,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40936 deadline: 1680987378669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:18,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:18,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40954 deadline: 1680987378672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:18,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:18,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40934 deadline: 1680987378673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:18,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680987318647, max timestamp=1680987318662Num uniques 1077; 2023-04-08 20:55:18,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=6870, suffix heap size=139976,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=150 2023-04-08 20:55:18,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680987318647, max timestamp=1680987318662Num uniques 1077; 2023-04-08 20:55:18,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=6870, suffix heap size=139976,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=150 2023-04-08 20:55:18,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680987318647, max timestamp=1680987318662Num uniques 1077; 2023-04-08 20:55:18,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=6870, suffix heap size=139976,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=150 2023-04-08 20:55:18,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680987318758, max timestamp=1680987318796Num uniques 1009; 2023-04-08 20:55:18,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=10806, suffix heap size=132904,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=236 2023-04-08 20:55:18,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680987318758, max timestamp=1680987318796Num uniques 1009; 2023-04-08 20:55:18,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=10806, suffix heap size=132904,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=236 2023-04-08 20:55:18,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680987318781, max timestamp=1680987318796Num uniques 1009; 2023-04-08 20:55:18,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-08 20:55:18,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680987318816, max timestamp=1680987318830Num uniques 1095; 2023-04-08 20:55:18,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-08 20:55:18,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680987318816, max timestamp=1680987318830Num uniques 1095; 2023-04-08 20:55:18,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-08 20:55:18,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680987318816, max timestamp=1680987318830Num uniques 1009; 2023-04-08 20:55:18,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=11220, suffix heap size=132904,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=245 2023-04-08 20:55:18,863 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:18,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40950 deadline: 1680987378857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:18,865 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:18,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40978 deadline: 1680987378859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:18,866 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:18,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40934 deadline: 1680987378861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:18,867 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:18,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40936 deadline: 1680987378861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:18,867 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:18,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40954 deadline: 1680987378863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:18,974 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:18,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40936 deadline: 1680987378973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:18,975 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:18,977 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/dd1d859c7c974090969b8f6c989e9843 2023-04-08 20:55:18,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40950 deadline: 1680987378973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:18,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:18,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40978 deadline: 1680987378974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:18,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:18,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40954 deadline: 1680987378973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:18,978 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:18,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40934 deadline: 1680987378977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,114 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/7a14eaf4f9694083800debf93ee163d9 2023-04-08 20:55:19,182 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:19,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40950 deadline: 1680987379182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,183 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:19,183 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:19,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40978 deadline: 1680987379183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,184 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/94ca5359441b46eb9e40c675dcb47629 2023-04-08 20:55:19,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40936 deadline: 1680987379182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:19,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:19,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40954 deadline: 1680987379185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40934 deadline: 1680987379185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,199 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/dd1d859c7c974090969b8f6c989e9843 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/dd1d859c7c974090969b8f6c989e9843 2023-04-08 20:55:19,212 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/dd1d859c7c974090969b8f6c989e9843, entries=150, sequenceid=9, filesize=11.6 K 2023-04-08 20:55:19,218 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/7a14eaf4f9694083800debf93ee163d9 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/7a14eaf4f9694083800debf93ee163d9 2023-04-08 20:55:19,242 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/7a14eaf4f9694083800debf93ee163d9, entries=150, sequenceid=9, filesize=11.6 K 2023-04-08 20:55:19,247 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/94ca5359441b46eb9e40c675dcb47629 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/94ca5359441b46eb9e40c675dcb47629 2023-04-08 20:55:19,268 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/94ca5359441b46eb9e40c675dcb47629, entries=150, sequenceid=9, filesize=11.6 K 2023-04-08 20:55:19,273 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 7f864e4ec7cff383a8a9db2371cff040 in 922ms, sequenceid=9, compaction requested=false 2023-04-08 20:55:19,275 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-08 20:55:19,278 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:19,278 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:19,278 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:19,279 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:19,279 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,279 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:19,279 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:19,284 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:19,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:19,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:19,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:19,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:19,287 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:19,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,288 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:19,288 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:19,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:19,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:19,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21d57e77[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:19,289 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:19,289 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:19,291 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:19,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:19,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:19,291 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:19,292 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:19,292 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:19,294 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:19,294 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:19,294 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:19,298 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,299 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:19,299 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:19,299 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:19,299 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:19,299 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:19,299 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:19,299 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:19,311 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:19,311 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:19,311 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:19,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:19,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:19,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:19,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:19,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:19,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:19,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:19,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:19,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:19,316 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:19,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,320 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:19,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:19,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,340 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,340 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:19,340 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:19,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:19,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:19,340 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:19,341 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,340 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:19,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:19,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:19,340 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:19,341 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:19,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:19,342 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:19,343 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:19,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:19,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:19,345 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:19,345 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:19,449 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:19,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:19,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:19,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:19,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:19,482 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:19,482 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:19,487 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:19,487 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:19,488 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:19,488 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:19,489 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:19,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:19,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:19,489 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:19,491 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,491 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:19,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:19,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:19,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:19,492 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:19,503 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7f864e4ec7cff383a8a9db2371cff040 3/3 column families, dataSize=154.31 KB heapSize=437.39 KB 2023-04-08 20:55:19,503 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=A 2023-04-08 20:55:19,504 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-08 20:55:19,504 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=B 2023-04-08 20:55:19,504 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-08 20:55:19,504 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=C 2023-04-08 20:55:19,504 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-08 20:55:19,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:19,514 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:19,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40978 deadline: 1680987379506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,520 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:19,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40954 deadline: 1680987379512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:19,522 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:19,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40934 deadline: 1680987379512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:19,522 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:19,522 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:19,523 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:19,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40950 deadline: 1680987379514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:19,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40936 deadline: 1680987379514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,527 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:19,527 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:19,527 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:19,527 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:19,527 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:19,530 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:19,530 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:19,550 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:19,550 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:19,550 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,550 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:19,550 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:19,553 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:19,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:19,554 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:19,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:19,555 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:19,555 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:19,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:19,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:19,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:19,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:19,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ad6874[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:19,558 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:19,558 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:19,568 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:19,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:19,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:19,575 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:19,575 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:19,575 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:19,575 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,575 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:19,579 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:19,579 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:19,579 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:19,579 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:19,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:19,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:19,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:19,583 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/38397fe5b0fd4bd8ac55392f617c5f29 2023-04-08 20:55:19,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:19,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:19,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:19,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:19,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:19,591 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:19,591 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:19,591 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:19,591 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:19,591 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:19,595 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:19,595 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:19,595 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:19,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:19,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:19,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:19,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:19,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:19,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:19,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:19,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:19,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:19,598 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:19,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:19,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:19,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,600 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:19,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:19,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,611 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,611 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:19,611 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:19,611 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:19,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:19,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:19,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:19,611 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:19,611 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:19,612 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,612 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:19,612 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:19,612 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:19,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:19,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:19,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:19,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:19,612 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:19,612 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:19,630 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/fe704aed132441ce8c5ff2dcf06a2511 2023-04-08 20:55:19,634 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:19,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40978 deadline: 1680987379618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,635 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:19,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40936 deadline: 1680987379630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,636 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:19,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:19,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40950 deadline: 1680987379631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40954 deadline: 1680987379633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,637 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:19,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40934 deadline: 1680987379633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,671 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/bfa7f40724304d84bf7c9210a24ffc08 2023-04-08 20:55:19,686 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/38397fe5b0fd4bd8ac55392f617c5f29 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/38397fe5b0fd4bd8ac55392f617c5f29 2023-04-08 20:55:19,696 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/38397fe5b0fd4bd8ac55392f617c5f29, entries=200, sequenceid=116, filesize=13.9 K 2023-04-08 20:55:19,699 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/fe704aed132441ce8c5ff2dcf06a2511 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/fe704aed132441ce8c5ff2dcf06a2511 2023-04-08 20:55:19,712 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:19,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:19,714 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/fe704aed132441ce8c5ff2dcf06a2511, entries=150, sequenceid=116, filesize=11.6 K 2023-04-08 20:55:19,718 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/bfa7f40724304d84bf7c9210a24ffc08 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/bfa7f40724304d84bf7c9210a24ffc08 2023-04-08 20:55:19,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:19,730 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/bfa7f40724304d84bf7c9210a24ffc08, entries=150, sequenceid=116, filesize=11.6 K 2023-04-08 20:55:19,731 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~437.34 KB/447840, currentSize=26.84 KB/27480 for 7f864e4ec7cff383a8a9db2371cff040 in 228ms, sequenceid=116, compaction requested=false 2023-04-08 20:55:19,733 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:19,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:19,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:19,734 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:19,734 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:19,735 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:19,735 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:19,736 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:19,736 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:19,737 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:19,737 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:19,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:19,737 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,738 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:19,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:19,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:19,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:19,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:19,739 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:19,739 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:19,739 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:19,742 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:19,742 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:19,748 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:19,748 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:19,748 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:19,751 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:19,751 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:19,752 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7f864e4ec7cff383a8a9db2371cff040 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-08 20:55:19,752 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=A 2023-04-08 20:55:19,753 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:19,753 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=B 2023-04-08 20:55:19,753 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:19,753 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=C 2023-04-08 20:55:19,753 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:19,807 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/817db19b718a487fa825800605da39f8 2023-04-08 20:55:19,856 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:19,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:19,858 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/5ab9cf0d2c294c538b1509859b8e2cdb 2023-04-08 20:55:19,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1680987319874, max timestamp=1680987319888Num uniques 859; 2023-04-08 20:55:19,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-08 20:55:19,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680987319874, max timestamp=1680987319888Num uniques 859; 2023-04-08 20:55:19,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-08 20:55:19,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680987319874, max timestamp=1680987319888Num uniques 859; 2023-04-08 20:55:19,907 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-08 20:55:19,910 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/0813ffcd2c724bda8ffb3627e1435e8d 2023-04-08 20:55:19,922 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/817db19b718a487fa825800605da39f8 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/817db19b718a487fa825800605da39f8 2023-04-08 20:55:19,933 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/817db19b718a487fa825800605da39f8, entries=150, sequenceid=123, filesize=11.6 K 2023-04-08 20:55:19,935 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/5ab9cf0d2c294c538b1509859b8e2cdb as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/5ab9cf0d2c294c538b1509859b8e2cdb 2023-04-08 20:55:19,959 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/5ab9cf0d2c294c538b1509859b8e2cdb, entries=150, sequenceid=123, filesize=11.6 K 2023-04-08 20:55:19,961 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/0813ffcd2c724bda8ffb3627e1435e8d as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/0813ffcd2c724bda8ffb3627e1435e8d 2023-04-08 20:55:19,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:19,968 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1680987319919, max timestamp=1680987319961Num uniques 1118; 2023-04-08 20:55:19,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1680987319919, max timestamp=1680987319961Num uniques 1118; 2023-04-08 20:55:19,968 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1680987319919, max timestamp=1680987319961Num uniques 1118; 2023-04-08 20:55:19,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=14558, suffix heap size=144240,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=318 2023-04-08 20:55:19,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=14558, suffix heap size=144240,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=318 2023-04-08 20:55:19,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=14558, suffix heap size=144240,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=318 2023-04-08 20:55:19,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40954 deadline: 1680987379937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,989 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/0813ffcd2c724bda8ffb3627e1435e8d, entries=150, sequenceid=123, filesize=11.6 K 2023-04-08 20:55:19,991 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=73.80 KB/75570 for 7f864e4ec7cff383a8a9db2371cff040 in 239ms, sequenceid=123, compaction requested=true 2023-04-08 20:55:19,991 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:19,991 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:19,991 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:19,991 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:19,991 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:19,991 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7f864e4ec7cff383a8a9db2371cff040 3/3 column families, dataSize=87.22 KB heapSize=230.63 KB 2023-04-08 20:55:19,992 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=A 2023-04-08 20:55:19,992 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:55:19,992 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=B 2023-04-08 20:55:19,992 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:19,992 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:19,993 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:55:19,993 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=C 2023-04-08 20:55:19,993 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:55:19,996 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,996 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:19,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:19,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:19,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:19,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:19,998 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:19,998 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:20,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:20,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:20,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:20,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:20,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74fd37f3[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:20,007 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:20,007 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,008 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,008 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:20,008 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:20,008 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:20,009 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,009 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:20,011 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:20,011 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:20,011 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:20,011 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:20,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:20,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:20,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:20,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:20,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:20,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:20,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:20,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:20,019 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:20,019 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:20,019 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:20,019 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:20,019 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:20,024 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,024 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:20,024 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:20,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:20,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:20,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:20,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:20,026 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:20,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:20,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:20,029 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:20,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:20,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,049 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:20,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40934 deadline: 1680987380042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,052 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:20,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40950 deadline: 1680987380047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,056 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,056 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:20,056 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:20,056 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:20,056 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:20,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:20,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:20,056 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:20,062 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,062 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,062 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,062 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:20,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:20,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:20,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:20,064 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:20,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:20,062 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:20,064 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:20,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40936 deadline: 1680987380049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40978 deadline: 1680987380050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,083 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:20,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40954 deadline: 1680987380077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,152 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:20,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40934 deadline: 1680987380151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:20,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40950 deadline: 1680987380155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,165 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:20,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:20,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:20,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:20,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40978 deadline: 1680987380170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,175 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:20,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:20,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:20,179 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:20,179 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:20,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40936 deadline: 1680987380170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,181 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:20,181 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:20,181 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,181 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:20,182 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:20,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:20,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:20,183 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,184 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,184 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:20,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:20,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:20,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:20,186 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:20,187 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:20,187 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:20,187 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:20,187 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:20,188 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:20,188 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:20,189 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:20,189 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:20,189 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:20,189 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:20,189 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:20,190 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:20,190 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:20,190 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,190 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:20,190 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:20,199 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,199 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:20,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:20,200 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,200 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:20,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:20,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:20,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:20,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:20,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:20,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73455b8f[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:20,204 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:20,204 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,208 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,208 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:20,208 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:20,208 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:20,208 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,208 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:20,211 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:20,211 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,211 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:20,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:20,211 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:20,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:20,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:20,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:20,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:20,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:20,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:20,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:20,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:20,222 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:20,222 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:20,222 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:20,222 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:20,222 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:20,224 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,224 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,224 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:20,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:20,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:20,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,225 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:20,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:20,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:20,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:20,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:20,228 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:20,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:20,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,242 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,242 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:20,242 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:20,242 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:20,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:20,242 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:20,243 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,243 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,243 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:20,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:20,243 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:20,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:20,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:20,250 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,251 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:20,251 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:20,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:20,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:20,287 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:20,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40954 deadline: 1680987380286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,351 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:20,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:20,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:20,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:20,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:20,357 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:20,357 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:20,358 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:20,359 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:20,360 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,360 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:20,361 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:20,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40934 deadline: 1680987380356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,361 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:20,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:20,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:20,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:20,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:20,363 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:20,363 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:20,363 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,363 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:20,363 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:20,363 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:20,363 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:20,364 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:20,364 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:20,365 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:20,365 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:20,365 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:20,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40950 deadline: 1680987380361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,366 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:20,366 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:20,366 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:20,366 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:20,367 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:20,367 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:20,367 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,368 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:20,368 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:20,375 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:20,375 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:20,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:20,376 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,376 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:20,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:20,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:20,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:20,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:20,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7969b556[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:20,378 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:20,378 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,383 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,383 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:20,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40978 deadline: 1680987380381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,384 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:20,384 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:20,384 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:20,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:20,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40936 deadline: 1680987380384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,391 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:20,391 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:20,391 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:20,391 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,393 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:20,393 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:20,393 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:20,393 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:20,394 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:20,394 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:20,394 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:20,399 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,400 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:20,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:20,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:20,401 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:20,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:20,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:20,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:20,405 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:20,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:20,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:20,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:20,409 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,413 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,413 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:20,413 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:20,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:20,413 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:20,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:20,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:20,413 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:20,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:20,415 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:20,415 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,415 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,415 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,415 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,416 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:20,416 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:20,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:20,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:20,448 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/82314af31af742a48041f3ea33f7cdb8 2023-04-08 20:55:20,500 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/f8dde369eedd4faa9828456fde8dd63e 2023-04-08 20:55:20,516 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:20,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:20,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:20,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:20,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:20,530 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:20,530 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:20,545 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:20,546 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:20,547 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,547 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:20,549 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:20,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:20,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:20,550 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:20,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:20,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:20,551 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:20,552 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,552 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:20,553 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:20,554 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:20,555 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:20,555 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:20,556 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:20,560 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:20,560 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:20,560 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:20,560 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:20,560 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:20,560 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:20,567 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:20,567 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:20,567 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,567 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:20,567 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:20,570 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/7fa021fe51494aae8eca0d0a5d5de938 2023-04-08 20:55:20,570 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:20,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:20,571 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,579 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,579 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:20,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:20,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:20,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:20,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:20,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:20,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@146a9cc7[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:20,582 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:20,582 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,583 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,583 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:20,583 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:20,583 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:20,584 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,584 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:20,585 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:20,585 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:20,585 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:20,585 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:20,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:20,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:20,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:20,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:20,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:20,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:20,593 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/82314af31af742a48041f3ea33f7cdb8 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/82314af31af742a48041f3ea33f7cdb8 2023-04-08 20:55:20,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:20,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:20,596 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:20,596 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:20,596 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:20,596 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:20,596 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:20,599 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,599 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,599 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:20,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:20,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:20,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:20,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:20,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:20,601 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:20,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:20,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:20,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,603 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:20,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:20,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:20,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40954 deadline: 1680987380595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,608 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:20,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:20,608 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,608 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:20,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:20,608 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/82314af31af742a48041f3ea33f7cdb8, entries=250, sequenceid=167, filesize=16.4 K 2023-04-08 20:55:20,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:20,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:20,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:20,609 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:20,609 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:20,608 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:20,608 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:20,609 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:20,609 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,609 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:20,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:20,609 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,609 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,612 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/f8dde369eedd4faa9828456fde8dd63e as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/f8dde369eedd4faa9828456fde8dd63e 2023-04-08 20:55:20,627 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/f8dde369eedd4faa9828456fde8dd63e, entries=150, sequenceid=167, filesize=11.8 K 2023-04-08 20:55:20,631 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/7fa021fe51494aae8eca0d0a5d5de938 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/7fa021fe51494aae8eca0d0a5d5de938 2023-04-08 20:55:20,642 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/7fa021fe51494aae8eca0d0a5d5de938, entries=150, sequenceid=167, filesize=11.8 K 2023-04-08 20:55:20,644 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~230.58 KB/236112, currentSize=107.34 KB/109920 for 7f864e4ec7cff383a8a9db2371cff040 in 653ms, sequenceid=167, compaction requested=true 2023-04-08 20:55:20,645 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:20,647 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:20,647 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:20,647 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:20,648 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 20:55:20,648 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 20:55:20,658 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47694 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 20:55:20,659 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54814 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 20:55:20,661 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 7f864e4ec7cff383a8a9db2371cff040/B is initiating minor compaction (all files) 2023-04-08 20:55:20,661 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): 7f864e4ec7cff383a8a9db2371cff040/A is initiating minor compaction (all files) 2023-04-08 20:55:20,661 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f864e4ec7cff383a8a9db2371cff040/B in TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:20,661 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f864e4ec7cff383a8a9db2371cff040/A in TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:20,661 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/dd1d859c7c974090969b8f6c989e9843, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/38397fe5b0fd4bd8ac55392f617c5f29, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/817db19b718a487fa825800605da39f8, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/82314af31af742a48041f3ea33f7cdb8] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp, totalSize=53.5 K 2023-04-08 20:55:20,661 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/7a14eaf4f9694083800debf93ee163d9, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/fe704aed132441ce8c5ff2dcf06a2511, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/5ab9cf0d2c294c538b1509859b8e2cdb, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/f8dde369eedd4faa9828456fde8dd63e] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp, totalSize=46.6 K 2023-04-08 20:55:20,664 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 7a14eaf4f9694083800debf93ee163d9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1680987318284 2023-04-08 20:55:20,664 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting dd1d859c7c974090969b8f6c989e9843, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1680987318284 2023-04-08 20:55:20,667 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting fe704aed132441ce8c5ff2dcf06a2511, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1680987318851 2023-04-08 20:55:20,669 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 38397fe5b0fd4bd8ac55392f617c5f29, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1680987318851 2023-04-08 20:55:20,670 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 5ab9cf0d2c294c538b1509859b8e2cdb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1680987319505 2023-04-08 20:55:20,671 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 817db19b718a487fa825800605da39f8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1680987319505 2023-04-08 20:55:20,672 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7f864e4ec7cff383a8a9db2371cff040 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-08 20:55:20,672 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting f8dde369eedd4faa9828456fde8dd63e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1680987319962 2023-04-08 20:55:20,672 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=A 2023-04-08 20:55:20,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:20,672 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:20,672 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=B 2023-04-08 20:55:20,672 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:20,672 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=C 2023-04-08 20:55:20,673 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:20,672 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 82314af31af742a48041f3ea33f7cdb8, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1680987319962 2023-04-08 20:55:20,710 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:20,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:20,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:20,726 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:20,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40950 deadline: 1680987380714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:20,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:20,730 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:20,730 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:20,731 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:20,731 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:20,732 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,732 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:20,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:20,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40978 deadline: 1680987380724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:20,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40934 deadline: 1680987380726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:20,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40936 deadline: 1680987380727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,739 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,739 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:20,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:20,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:20,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:20,740 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7f864e4ec7cff383a8a9db2371cff040#A#compaction#13 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:20,741 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 7f864e4ec7cff383a8a9db2371cff040#B#compaction#14 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:20,741 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,742 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:20,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:20,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:20,742 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:20,754 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:20,754 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:20,754 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:20,754 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:20,758 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:20,758 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:20,758 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:20,759 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:20,759 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:20,759 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:20,759 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:20,763 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:20,763 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:20,763 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,763 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:20,763 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:20,765 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,765 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:20,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:20,765 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,765 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:20,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:20,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:20,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:20,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:20,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:20,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9356316[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:20,774 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:20,774 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,784 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,784 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:20,784 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,785 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:20,784 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:20,785 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:20,787 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:20,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:20,788 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:20,788 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:20,788 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:20,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:20,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:20,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:20,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:20,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:20,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:20,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:20,803 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:20,803 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:20,803 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:20,804 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:20,804 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:20,808 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,808 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:20,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:20,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:20,809 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:20,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:20,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,810 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/610441322c114074b0bb9bb3a69580fb as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/610441322c114074b0bb9bb3a69580fb 2023-04-08 20:55:20,810 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:20,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:20,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:20,815 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/ef6f9dff16bc4e1b8785f9ceea7c60ed as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/ef6f9dff16bc4e1b8785f9ceea7c60ed 2023-04-08 20:55:20,815 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:20,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:20,827 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:20,827 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:20,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:20,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:20,827 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,827 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:20,827 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:20,827 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,827 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,827 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,827 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,828 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:20,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:20,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:20,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:20,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:20,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:20,830 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:20,830 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:20,832 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:20,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40950 deadline: 1680987380829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,839 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:20,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40978 deadline: 1680987380837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:20,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40936 deadline: 1680987380837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,841 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:20,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40934 deadline: 1680987380838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,843 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7f864e4ec7cff383a8a9db2371cff040/B of 7f864e4ec7cff383a8a9db2371cff040 into ef6f9dff16bc4e1b8785f9ceea7c60ed(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:20,843 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7f864e4ec7cff383a8a9db2371cff040/A of 7f864e4ec7cff383a8a9db2371cff040 into 610441322c114074b0bb9bb3a69580fb(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:20,844 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:20,844 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:20,844 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040., storeName=7f864e4ec7cff383a8a9db2371cff040/B, priority=12, startTime=1680987320647; duration=0sec 2023-04-08 20:55:20,844 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040., storeName=7f864e4ec7cff383a8a9db2371cff040/A, priority=12, startTime=1680987320647; duration=0sec 2023-04-08 20:55:20,845 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:20,845 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:20,845 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 20:55:20,850 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47694 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 20:55:20,851 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): 7f864e4ec7cff383a8a9db2371cff040/C is initiating minor compaction (all files) 2023-04-08 20:55:20,851 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f864e4ec7cff383a8a9db2371cff040/C in TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:20,851 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/94ca5359441b46eb9e40c675dcb47629, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/bfa7f40724304d84bf7c9210a24ffc08, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/0813ffcd2c724bda8ffb3627e1435e8d, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/7fa021fe51494aae8eca0d0a5d5de938] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp, totalSize=46.6 K 2023-04-08 20:55:20,852 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 94ca5359441b46eb9e40c675dcb47629, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1680987318284 2023-04-08 20:55:20,855 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting bfa7f40724304d84bf7c9210a24ffc08, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1680987318851 2023-04-08 20:55:20,859 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 0813ffcd2c724bda8ffb3627e1435e8d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1680987319505 2023-04-08 20:55:20,860 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 7fa021fe51494aae8eca0d0a5d5de938, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1680987319962 2023-04-08 20:55:20,885 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7f864e4ec7cff383a8a9db2371cff040#C#compaction#15 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:20,930 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:20,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:20,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:20,937 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/c2bb36811fbc407cb1d0e50f718f880f as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/c2bb36811fbc407cb1d0e50f718f880f 2023-04-08 20:55:20,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:20,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:20,942 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:20,946 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:20,947 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:20,947 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:20,948 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,948 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:20,954 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:20,954 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:20,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:20,955 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,955 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:20,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:20,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:20,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:20,956 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:20,958 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:20,958 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:20,958 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:20,958 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:20,959 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:20,966 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:20,966 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:20,966 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:20,968 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:20,968 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:20,968 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:20,968 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:20,969 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:20,969 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,969 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:20,969 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:20,970 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,970 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:20,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:20,971 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,971 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:20,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:20,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:20,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:20,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:20,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:20,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50f5c89c[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:20,973 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:20,973 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,975 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,975 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7f864e4ec7cff383a8a9db2371cff040/C of 7f864e4ec7cff383a8a9db2371cff040 into c2bb36811fbc407cb1d0e50f718f880f(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:20,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:20,975 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:20,975 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:20,975 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:20,975 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:20,975 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040., storeName=7f864e4ec7cff383a8a9db2371cff040/C, priority=12, startTime=1680987320647; duration=0sec 2023-04-08 20:55:20,975 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:20,977 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:20,977 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:20,977 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:20,978 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,979 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:20,979 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:20,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:20,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:20,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:20,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:20,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:20,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:20,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:20,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:20,983 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:20,983 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:20,983 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:20,983 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:20,983 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:20,985 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,985 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,985 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:20,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:20,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:20,986 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:20,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:20,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:20,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:20,987 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:20,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:20,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:20,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:20,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:21,000 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,000 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:21,000 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:21,000 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:21,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:21,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:21,000 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,001 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,001 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,001 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,000 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:21,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:21,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:21,001 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:21,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:21,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:21,003 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:21,003 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:21,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:21,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:21,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40950 deadline: 1680987381034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,042 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:21,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40978 deadline: 1680987381041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,043 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:21,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40936 deadline: 1680987381042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,046 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:21,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40934 deadline: 1680987381045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,094 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-08 20:55:21,111 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:21,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:21,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40954 deadline: 1680987381111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:21,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:21,158 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/4e886adf443e4de48ac34650d680ee1b 2023-04-08 20:55:21,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:21,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:21,159 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:21,160 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:21,164 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:21,164 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:21,164 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,165 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:21,171 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:21,171 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:21,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:21,173 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,173 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:21,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:21,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:21,175 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:21,178 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:21,178 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:21,178 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:21,178 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:21,179 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:21,182 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:21,182 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:21,190 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:21,190 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:21,190 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:21,191 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:21,191 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:21,191 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:21,191 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,191 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:21,191 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:21,192 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,192 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:21,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:21,193 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,193 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:21,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:21,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:21,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:21,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:21,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:21,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@640214d1[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:21,195 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:21,198 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,199 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,200 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:21,200 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:21,200 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:21,200 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,200 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:21,201 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,201 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:21,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:21,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:21,201 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:21,202 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:21,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:21,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:21,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:21,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:21,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:21,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:21,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:21,231 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:21,232 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:21,232 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:21,232 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:21,232 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:21,234 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,234 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:21,234 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:21,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:21,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:21,235 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:21,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,238 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:21,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:21,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,252 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:21,252 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:21,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:21,252 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:21,255 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:21,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:21,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:21,252 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:21,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:21,255 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:21,255 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:21,255 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:21,255 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,255 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,256 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,256 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:21,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:21,273 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/ca911e4d57944ab593bddc555c51fc6c 2023-04-08 20:55:21,284 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-08 20:55:21,285 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-08 20:55:21,339 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:21,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40950 deadline: 1680987381337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,347 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:21,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40936 deadline: 1680987381347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,349 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:21,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40978 deadline: 1680987381347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,354 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:21,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40934 deadline: 1680987381353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,355 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:21,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:21,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:21,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:21,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:21,360 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:21,360 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:21,362 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:21,363 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:21,363 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,363 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:21,364 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:21,364 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:21,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:21,364 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,364 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:21,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:21,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:21,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:21,365 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:21,365 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:21,365 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:21,366 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:21,366 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:21,368 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:21,368 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:21,368 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:21,368 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:21,368 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:21,368 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:21,368 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:21,368 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:21,368 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:21,368 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,368 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:21,368 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:21,374 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,374 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:21,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:21,375 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,375 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:21,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:21,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:21,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:21,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:21,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:21,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@244e2032[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:21,377 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:21,377 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,382 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,383 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:21,383 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:21,383 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:21,383 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,383 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:21,384 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,384 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:21,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:21,384 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:21,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:21,384 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:21,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:21,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:21,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:21,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:21,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:21,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:21,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:21,390 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:21,390 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:21,390 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:21,390 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:21,390 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:21,399 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,399 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:21,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:21,399 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:21,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:21,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:21,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:21,401 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:21,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:21,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:21,403 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:21,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,426 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,427 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:21,427 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:21,427 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:21,427 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,427 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,427 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:21,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:21,427 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:21,427 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:21,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:21,427 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:21,427 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:21,427 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:21,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:21,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:21,527 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:21,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:21,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:21,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:21,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:21,532 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:21,532 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:21,533 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:21,533 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:21,534 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,534 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:21,535 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:21,535 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:21,535 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:21,535 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:21,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:21,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:21,536 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:21,537 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:21,537 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:21,537 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:21,538 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:21,538 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:21,538 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:21,538 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:21,538 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:21,538 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:21,538 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:21,538 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:21,538 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:21,539 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,539 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:21,539 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:21,540 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,540 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:21,541 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,541 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:21,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:21,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:21,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:21,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:21,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:21,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@641da7e[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:21,543 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:21,543 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,544 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,545 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,545 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:21,546 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:21,546 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:21,546 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:21,548 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:21,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:21,549 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:21,549 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:21,549 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:21,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:21,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:21,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:21,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:21,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:21,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:21,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:21,562 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:21,562 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:21,562 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:21,562 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:21,562 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:21,566 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,566 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:21,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,567 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:21,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:21,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:21,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:21,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:21,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,569 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:21,571 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:21,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:21,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:21,579 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:21,579 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:21,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:21,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:21,580 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:21,580 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:21,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:21,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:21,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:21,580 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:21,584 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:21,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:21,584 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:21,585 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:21,585 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,585 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,585 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,585 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,685 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:21,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:21,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:21,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:21,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:21,691 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:21,691 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:21,692 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:21,692 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:21,693 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,693 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:21,696 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,696 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:21,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:21,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:21,697 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,697 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:21,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:21,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:21,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:21,698 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:21,698 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:21,698 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:21,698 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:21,698 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:21,699 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:21,706 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:21,706 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:21,706 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:21,706 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:21,706 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:21,706 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:21,707 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:21,707 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:21,707 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,707 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:21,707 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:21,708 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,708 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:21,709 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,710 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:21,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:21,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:21,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:21,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:21,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:21,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@788e4302[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:21,712 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:21,712 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,713 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,713 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:21,713 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:21,713 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:21,713 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,713 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:21,714 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:21,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:21,714 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:21,714 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:21,714 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:21,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:21,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:21,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:21,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:21,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:21,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:21,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:21,718 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:21,718 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:21,718 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:21,718 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:21,718 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:21,720 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/7d11d98db0f04c30a525c42b67812a2f 2023-04-08 20:55:21,723 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,723 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:21,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,724 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:21,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:21,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:21,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:21,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:21,727 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:21,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:21,729 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:21,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:21,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:21,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,733 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/4e886adf443e4de48ac34650d680ee1b as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/4e886adf443e4de48ac34650d680ee1b 2023-04-08 20:55:21,741 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:21,742 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:21,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:21,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:21,742 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,742 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:21,742 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:21,742 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,742 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,742 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,742 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:21,742 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:21,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:21,744 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:21,744 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:21,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:21,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:21,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:21,747 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/4e886adf443e4de48ac34650d680ee1b, entries=200, sequenceid=187, filesize=14.1 K 2023-04-08 20:55:21,754 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/ca911e4d57944ab593bddc555c51fc6c as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/ca911e4d57944ab593bddc555c51fc6c 2023-04-08 20:55:21,765 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/ca911e4d57944ab593bddc555c51fc6c, entries=150, sequenceid=187, filesize=11.8 K 2023-04-08 20:55:21,769 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/7d11d98db0f04c30a525c42b67812a2f as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/7d11d98db0f04c30a525c42b67812a2f 2023-04-08 20:55:21,776 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/7d11d98db0f04c30a525c42b67812a2f, entries=150, sequenceid=187, filesize=11.8 K 2023-04-08 20:55:21,781 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 7f864e4ec7cff383a8a9db2371cff040 in 1109ms, sequenceid=187, compaction requested=false 2023-04-08 20:55:21,782 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:21,844 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:21,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:21,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:21,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:21,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:21,849 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:21,849 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:21,850 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:21,850 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:21,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:21,851 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,851 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:21,851 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7f864e4ec7cff383a8a9db2371cff040 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-08 20:55:21,851 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=A 2023-04-08 20:55:21,851 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:21,851 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=B 2023-04-08 20:55:21,852 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:21,852 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=C 2023-04-08 20:55:21,852 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:21,852 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:21,852 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:21,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:21,852 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:21,852 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:21,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:21,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:21,853 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:21,853 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:21,854 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:21,854 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:21,854 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:21,858 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:21,858 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:21,858 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:21,858 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:21,858 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:21,858 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:21,859 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:21,859 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:21,859 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:21,859 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,859 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:21,859 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:21,866 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,866 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:21,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:21,866 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,866 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:21,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:21,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:21,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:21,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:21,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:21,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cde745e[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:21,872 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:21,872 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,878 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,878 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,878 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:21,878 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:21,879 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:21,879 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:21,883 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:21,883 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:21,883 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:21,883 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:21,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:21,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:21,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:21,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:21,889 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:21,889 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:21,889 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:21,889 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:21,889 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:21,891 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,891 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:21,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:21,891 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:21,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,892 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:21,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:21,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:21,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:21,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:21,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:21,894 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:21,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:21,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,899 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:21,899 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:21,899 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,900 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:21,900 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:21,900 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,900 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,900 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:21,900 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:21,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:21,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:21,900 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:21,900 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:21,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:21,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:21,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:21,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:21,900 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:21,900 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:21,908 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:21,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40936 deadline: 1680987381893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,908 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:21,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40950 deadline: 1680987381901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,909 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:21,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40978 deadline: 1680987381907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:21,909 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:21,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40934 deadline: 1680987381907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,000 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:22,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:22,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:22,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:22,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:22,006 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:22,006 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:22,007 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:22,007 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:22,007 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,007 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:22,008 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,008 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:22,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:22,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:22,009 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:22,009 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:22,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:22,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:22,010 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:22,010 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:22,010 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:22,011 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:22,011 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:22,012 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:22,012 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:22,012 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:22,012 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:22,012 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:22,012 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:22,012 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:22,013 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:22,013 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:22,013 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:22,013 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40978 deadline: 1680987382010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,013 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:22,013 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:22,013 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:22,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40936 deadline: 1680987382010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,013 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:22,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40950 deadline: 1680987382010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,014 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:22,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40934 deadline: 1680987382012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,016 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,016 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:22,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:22,016 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,016 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:22,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:22,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:22,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:22,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:22,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:22,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59c49a47[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:22,018 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:22,018 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,019 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,019 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,019 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:22,019 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:22,019 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:22,019 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:22,020 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:22,020 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,020 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:22,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:22,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:22,021 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:22,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:22,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:22,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:22,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:22,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:22,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:22,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:22,024 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:22,024 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:22,024 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:22,024 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:22,024 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:22,025 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,025 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:22,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,025 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:22,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:22,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:22,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:22,026 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:22,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:22,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:22,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:22,027 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:22,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:22,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,030 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:22,030 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:22,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:22,030 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:22,030 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:22,031 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:22,031 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,031 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:22,031 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:22,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:22,034 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,035 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,035 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:22,035 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:22,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:22,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:22,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:22,119 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:22,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40954 deadline: 1680987382117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,135 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:22,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:22,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:22,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:22,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:22,146 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:22,146 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:22,147 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:22,147 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:22,148 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,148 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:22,149 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,149 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:22,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:22,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:22,149 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,149 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:22,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:22,151 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:22,162 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:22,162 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:22,162 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:22,162 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:22,165 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:22,166 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:22,167 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:22,168 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:22,168 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:22,168 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:22,168 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:22,168 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:22,168 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:22,168 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,168 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:22,168 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:22,170 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,170 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:22,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:22,170 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,170 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:22,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:22,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:22,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:22,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:22,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:22,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@515e392b[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:22,171 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:22,172 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,173 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,173 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:22,173 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,173 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:22,173 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:22,173 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:22,175 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:22,175 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:22,175 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,175 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:22,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:22,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:22,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:22,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:22,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:22,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:22,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:22,179 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:22,179 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:22,179 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:22,179 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:22,179 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:22,180 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,180 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,181 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:22,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:22,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:22,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:22,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:22,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:22,183 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:22,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:22,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:22,187 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:22,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:22,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,190 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:22,190 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:22,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:22,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:22,191 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,191 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:22,191 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:22,191 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,191 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:22,191 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,191 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:22,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:22,191 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:22,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:22,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:22,195 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:22,195 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:22,217 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:22,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40978 deadline: 1680987382216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,217 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:22,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40934 deadline: 1680987382216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,218 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:22,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40950 deadline: 1680987382216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,218 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:22,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40936 deadline: 1680987382217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,279 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=204 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/64490481dcbd46d6bc6158e839c52142 2023-04-08 20:55:22,296 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:22,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:22,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:22,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:22,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:22,305 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:22,305 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:22,306 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:22,307 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:22,308 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,308 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:22,312 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:22,312 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:22,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:22,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:22,315 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,315 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:22,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:22,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:22,316 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:22,317 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:22,317 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:22,317 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:22,317 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:22,318 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:22,319 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:22,319 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:22,319 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:22,319 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:22,319 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:22,319 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:22,319 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:22,319 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:22,319 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,319 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:22,319 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:22,324 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,324 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:22,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:22,325 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,325 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:22,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:22,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:22,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:22,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:22,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:22,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bc2e651[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:22,331 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:22,331 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,333 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,333 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:22,333 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:22,333 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:22,333 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,333 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:22,334 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:22,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:22,334 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:22,334 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:22,334 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:22,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:22,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:22,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:22,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:22,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:22,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:22,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:22,347 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:22,347 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:22,347 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:22,347 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:22,347 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:22,352 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,352 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,352 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:22,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:22,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:22,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:22,352 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:22,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:22,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:22,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:22,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:22,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,359 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:22,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:22,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,361 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:22,361 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:22,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:22,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:22,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:22,361 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:22,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:22,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:22,363 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:22,363 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:22,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:22,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:22,364 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,364 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:22,364 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:22,364 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,364 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,364 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,364 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,463 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:22,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:22,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:22,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:22,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:22,472 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:22,472 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:22,473 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:22,473 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:22,474 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,474 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:22,474 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:22,474 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:22,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:22,475 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,475 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:22,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:22,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:22,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:22,476 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:22,482 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:22,482 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:22,482 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:22,482 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:22,486 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:22,487 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:22,487 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:22,487 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:22,487 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:22,487 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:22,487 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:22,487 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:22,487 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:22,487 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,487 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:22,487 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:22,491 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,491 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:22,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:22,491 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:22,491 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:22,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:22,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:22,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:22,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:22,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24c105f3[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:22,492 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:22,493 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,493 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,493 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:22,493 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:22,494 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:22,494 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,494 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:22,494 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:22,494 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:22,494 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,495 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:22,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:22,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:22,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:22,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:22,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:22,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:22,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:22,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:22,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:22,500 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:22,500 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:22,500 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:22,500 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:22,500 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:22,507 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,507 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,507 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:22,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:22,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:22,508 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:22,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:22,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:22,509 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:22,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:22,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:22,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:22,516 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:22,516 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:22,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:22,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:22,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:22,516 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,517 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:22,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:22,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:22,517 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:22,519 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,519 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,519 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,519 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,519 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:22,519 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:22,520 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:22,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40978 deadline: 1680987382520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,521 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:22,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40934 deadline: 1680987382521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,525 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:22,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:22,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40936 deadline: 1680987382522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40950 deadline: 1680987382523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,619 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:22,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:22,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:22,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:22,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:22,626 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:22,626 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:22,626 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:22,626 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:22,627 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,627 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:22,628 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:22,628 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:22,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:22,628 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,628 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:22,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:22,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:22,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:22,629 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:22,630 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:22,630 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:22,630 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:22,630 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:22,631 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:22,631 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:22,631 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:22,631 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:22,631 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:22,631 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:22,631 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:22,631 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:22,631 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:22,631 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,631 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:22,631 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:22,632 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,632 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:22,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:22,633 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,633 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:22,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:22,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:22,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:22,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:22,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:22,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@592aa58e[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:22,634 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:22,634 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,635 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,635 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,635 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:22,635 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:22,635 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:22,635 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:22,637 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:22,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:22,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:22,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:22,638 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:22,638 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:22,638 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:22,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:22,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:22,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:22,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:22,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:22,641 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:22,641 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:22,641 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:22,641 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:22,641 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:22,642 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,642 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:22,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,643 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:22,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:22,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:22,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:22,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:22,644 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:22,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:22,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:22,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,645 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:22,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:22,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,647 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:22,648 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:22,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:22,648 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:22,648 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:22,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:22,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:22,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:22,648 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:22,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:22,648 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:22,650 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:22,651 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,651 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:22,651 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,651 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,651 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,714 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-08 20:55:22,714 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-08 20:55:22,723 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=204 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/91a869ef29d044419264d73786666285 2023-04-08 20:55:22,751 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:22,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:22,755 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=204 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/c32c2d2a11814da792fc1eed9ce925fd 2023-04-08 20:55:22,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:22,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:22,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:22,758 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:22,758 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:22,759 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:22,759 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:22,761 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,761 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:22,762 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:22,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:22,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:22,762 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:22,763 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,763 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:22,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:22,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:22,764 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:22,765 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:22,765 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:22,765 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:22,765 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:22,766 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:22,766 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:22,766 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:22,766 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:22,766 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:22,766 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:22,766 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:22,766 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:22,766 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:22,767 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,767 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:22,767 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:22,768 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:22,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:22,768 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,769 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,769 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:22,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:22,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:22,770 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/64490481dcbd46d6bc6158e839c52142 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/64490481dcbd46d6bc6158e839c52142 2023-04-08 20:55:22,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:22,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:22,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:22,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@799d58e1[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:22,778 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:22,778 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,783 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,784 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,784 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:22,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,784 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:22,785 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:22,785 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:22,786 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/64490481dcbd46d6bc6158e839c52142, entries=200, sequenceid=204, filesize=14.1 K 2023-04-08 20:55:22,788 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/91a869ef29d044419264d73786666285 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/91a869ef29d044419264d73786666285 2023-04-08 20:55:22,791 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:22,791 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,791 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:22,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,792 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:22,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:22,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:22,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:22,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:22,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:22,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:22,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:22,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:22,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:22,795 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:22,795 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:22,795 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:22,795 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:22,796 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:22,799 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,799 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:22,799 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,799 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/91a869ef29d044419264d73786666285, entries=150, sequenceid=204, filesize=11.8 K 2023-04-08 20:55:22,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:22,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:22,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:22,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:22,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:22,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:22,802 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,802 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/c32c2d2a11814da792fc1eed9ce925fd as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/c32c2d2a11814da792fc1eed9ce925fd 2023-04-08 20:55:22,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:22,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:22,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,803 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:22,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:22,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,807 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,807 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:22,807 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:22,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:22,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:22,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:22,808 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:22,808 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,808 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,808 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:22,808 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:22,808 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:22,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:22,807 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:22,807 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:22,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:22,808 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:22,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:22,812 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/c32c2d2a11814da792fc1eed9ce925fd, entries=150, sequenceid=204, filesize=11.8 K 2023-04-08 20:55:22,814 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 7f864e4ec7cff383a8a9db2371cff040 in 963ms, sequenceid=204, compaction requested=true 2023-04-08 20:55:22,814 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:22,814 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:22,814 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:55:22,814 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:55:22,814 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:22,814 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:22,818 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41026 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:55:22,819 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): 7f864e4ec7cff383a8a9db2371cff040/A is initiating minor compaction (all files) 2023-04-08 20:55:22,819 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f864e4ec7cff383a8a9db2371cff040/A in TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:22,819 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/610441322c114074b0bb9bb3a69580fb, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/4e886adf443e4de48ac34650d680ee1b, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/64490481dcbd46d6bc6158e839c52142] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp, totalSize=40.1 K 2023-04-08 20:55:22,819 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36246 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:55:22,820 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 7f864e4ec7cff383a8a9db2371cff040/B is initiating minor compaction (all files) 2023-04-08 20:55:22,820 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f864e4ec7cff383a8a9db2371cff040/B in TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:22,820 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/ef6f9dff16bc4e1b8785f9ceea7c60ed, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/ca911e4d57944ab593bddc555c51fc6c, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/91a869ef29d044419264d73786666285] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp, totalSize=35.4 K 2023-04-08 20:55:22,820 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 610441322c114074b0bb9bb3a69580fb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1680987319962 2023-04-08 20:55:22,821 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting ef6f9dff16bc4e1b8785f9ceea7c60ed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1680987319962 2023-04-08 20:55:22,823 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting ca911e4d57944ab593bddc555c51fc6c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1680987320035 2023-04-08 20:55:22,823 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 4e886adf443e4de48ac34650d680ee1b, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1680987320035 2023-04-08 20:55:22,823 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 91a869ef29d044419264d73786666285, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1680987320711 2023-04-08 20:55:22,823 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 64490481dcbd46d6bc6158e839c52142, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1680987320711 2023-04-08 20:55:22,835 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 7f864e4ec7cff383a8a9db2371cff040#B#compaction#21 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:22,838 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7f864e4ec7cff383a8a9db2371cff040#A#compaction#22 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:22,880 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/78779c6aea004a1e88fd4bfe778e3e47 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/78779c6aea004a1e88fd4bfe778e3e47 2023-04-08 20:55:22,880 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/f1e853fdff714a898e3b9ecddd7d4a0e as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/f1e853fdff714a898e3b9ecddd7d4a0e 2023-04-08 20:55:22,900 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7f864e4ec7cff383a8a9db2371cff040/A of 7f864e4ec7cff383a8a9db2371cff040 into 78779c6aea004a1e88fd4bfe778e3e47(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:22,901 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:22,901 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040., storeName=7f864e4ec7cff383a8a9db2371cff040/A, priority=13, startTime=1680987322814; duration=0sec 2023-04-08 20:55:22,901 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:22,901 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:55:22,901 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7f864e4ec7cff383a8a9db2371cff040/B of 7f864e4ec7cff383a8a9db2371cff040 into f1e853fdff714a898e3b9ecddd7d4a0e(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:22,902 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:22,902 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040., storeName=7f864e4ec7cff383a8a9db2371cff040/B, priority=13, startTime=1680987322814; duration=0sec 2023-04-08 20:55:22,903 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:22,906 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36246 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:55:22,907 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): 7f864e4ec7cff383a8a9db2371cff040/C is initiating minor compaction (all files) 2023-04-08 20:55:22,907 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f864e4ec7cff383a8a9db2371cff040/C in TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:22,907 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/c2bb36811fbc407cb1d0e50f718f880f, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/7d11d98db0f04c30a525c42b67812a2f, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/c32c2d2a11814da792fc1eed9ce925fd] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp, totalSize=35.4 K 2023-04-08 20:55:22,908 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:22,908 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting c2bb36811fbc407cb1d0e50f718f880f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1680987319962 2023-04-08 20:55:22,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:22,910 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 7d11d98db0f04c30a525c42b67812a2f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1680987320035 2023-04-08 20:55:22,915 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting c32c2d2a11814da792fc1eed9ce925fd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1680987320711 2023-04-08 20:55:22,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:22,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:22,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:22,920 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:22,920 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:22,921 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:22,921 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:22,922 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,922 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:22,923 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,923 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:22,923 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:22,924 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:22,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:22,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:22,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:22,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:22,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:22,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:22,926 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:22,926 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:22,926 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:22,926 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:22,926 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:22,928 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:22,929 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:22,929 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:22,929 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:22,931 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:22,931 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7f864e4ec7cff383a8a9db2371cff040 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-08 20:55:22,931 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=A 2023-04-08 20:55:22,931 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:22,931 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=B 2023-04-08 20:55:22,931 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:22,932 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=C 2023-04-08 20:55:22,932 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:22,932 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7f864e4ec7cff383a8a9db2371cff040#C#compaction#23 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:22,967 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=225 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/5f330c06c12045029afddaab062b12c6 2023-04-08 20:55:22,988 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=225 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/27557d97ffa4483c87ceedc332910c67 2023-04-08 20:55:23,022 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=225 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/3229064441c64a68b104f585dbfe3fc7 2023-04-08 20:55:23,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:23,028 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:23,033 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/5f330c06c12045029afddaab062b12c6 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/5f330c06c12045029afddaab062b12c6 2023-04-08 20:55:23,043 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/5f330c06c12045029afddaab062b12c6, entries=150, sequenceid=225, filesize=11.8 K 2023-04-08 20:55:23,047 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/27557d97ffa4483c87ceedc332910c67 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/27557d97ffa4483c87ceedc332910c67 2023-04-08 20:55:23,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:23,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40936 deadline: 1680987383043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,060 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/27557d97ffa4483c87ceedc332910c67, entries=150, sequenceid=225, filesize=11.8 K 2023-04-08 20:55:23,063 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/3229064441c64a68b104f585dbfe3fc7 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/3229064441c64a68b104f585dbfe3fc7 2023-04-08 20:55:23,067 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:23,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40950 deadline: 1680987383052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,068 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:23,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40978 deadline: 1680987383059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,070 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:23,074 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/3229064441c64a68b104f585dbfe3fc7, entries=150, sequenceid=225, filesize=11.8 K 2023-04-08 20:55:23,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40934 deadline: 1680987383067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,081 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 7f864e4ec7cff383a8a9db2371cff040 in 150ms, sequenceid=225, compaction requested=false 2023-04-08 20:55:23,082 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:23,082 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:23,082 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:23,082 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:23,082 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,082 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:23,082 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:23,083 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,083 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:23,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:23,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:23,084 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,084 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:23,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:23,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:23,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:23,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:23,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35a1c9ed[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:23,086 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:23,086 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,087 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,087 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:23,087 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:23,087 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:23,087 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,087 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:23,088 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:23,088 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:23,088 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,088 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:23,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:23,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:23,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:23,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:23,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:23,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:23,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:23,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:23,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:23,094 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:23,094 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:23,094 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:23,094 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:23,094 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:23,099 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,099 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,099 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:23,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:23,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:23,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:23,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,100 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:23,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:23,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:23,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:23,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:23,102 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:23,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:23,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,108 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:23,108 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:23,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:23,108 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:23,108 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:23,108 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:23,108 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,108 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,108 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,108 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,108 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:23,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:23,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:23,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:23,112 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:23,112 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:23,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:23,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:23,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:23,165 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7f864e4ec7cff383a8a9db2371cff040 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-08 20:55:23,165 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=A 2023-04-08 20:55:23,165 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:23,165 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=B 2023-04-08 20:55:23,165 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:23,165 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=C 2023-04-08 20:55:23,165 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:23,207 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:23,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40978 deadline: 1680987383204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:23,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40934 deadline: 1680987383204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:23,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40950 deadline: 1680987383206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,209 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:23,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40936 deadline: 1680987383207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,212 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:23,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:23,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:23,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:23,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:23,216 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:23,217 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:23,217 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:23,217 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:23,217 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,217 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:23,218 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:23,218 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:23,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:23,218 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,218 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:23,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:23,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:23,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:23,219 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:23,219 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:23,219 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:23,219 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:23,220 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:23,220 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:23,220 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:23,220 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:23,220 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:23,220 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:23,220 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:23,220 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:23,220 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:23,220 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:23,220 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,220 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:23,220 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:23,221 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,221 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:23,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:23,222 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,222 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:23,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:23,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:23,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:23,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:23,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:23,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19e3ac37[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:23,224 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:23,224 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,225 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,225 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:23,225 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:23,226 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:23,226 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:23,226 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:23,227 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:23,227 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,227 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:23,227 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:23,227 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:23,227 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:23,227 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:23,227 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:23,227 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:23,229 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,229 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:23,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,229 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:23,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:23,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:23,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:23,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:23,230 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:23,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:23,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:23,231 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:23,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:23,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,234 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:23,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:23,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:23,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:23,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:23,234 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:23,234 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:23,234 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:23,234 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:23,234 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:23,234 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:23,234 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:23,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:23,235 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:23,235 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,235 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,235 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,310 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:23,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40978 deadline: 1680987383309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:23,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40934 deadline: 1680987383309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:23,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40950 deadline: 1680987383309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:23,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40936 deadline: 1680987383311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,334 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:23,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:23,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:23,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:23,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:23,338 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:23,339 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:23,339 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:23,339 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:23,339 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,339 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:23,340 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:23,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:23,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:23,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:23,341 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,341 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,341 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:23,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:23,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:23,341 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:23,342 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:23,342 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:23,342 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:23,342 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:23,342 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:23,347 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:23,347 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:23,347 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:23,347 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:23,356 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:23,356 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:23,356 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:23,356 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:23,356 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,356 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:23,357 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:23,357 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/4cd9ff4a596a445cbc0f3e8f28a93c2c as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/4cd9ff4a596a445cbc0f3e8f28a93c2c 2023-04-08 20:55:23,358 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,358 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:23,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:23,361 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,361 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:23,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:23,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:23,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:23,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:23,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:23,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15e97c50[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:23,364 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:23,364 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,366 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,366 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,366 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:23,366 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:23,366 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:23,366 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:23,367 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,367 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:23,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:23,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:23,367 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:23,368 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:23,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:23,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:23,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:23,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:23,371 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7f864e4ec7cff383a8a9db2371cff040/C of 7f864e4ec7cff383a8a9db2371cff040 into 4cd9ff4a596a445cbc0f3e8f28a93c2c(size=12.0 K), total size for store is 23.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:23,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:23,371 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:23,371 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040., storeName=7f864e4ec7cff383a8a9db2371cff040/C, priority=13, startTime=1680987322814; duration=0sec 2023-04-08 20:55:23,371 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:23,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:23,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:23,372 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:23,372 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:23,372 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:23,372 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:23,373 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:23,382 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,382 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:23,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:23,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,383 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:23,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:23,383 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:23,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:23,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:23,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:23,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:23,385 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:23,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:23,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,391 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:23,391 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,391 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:23,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:23,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:23,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:23,391 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:23,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:23,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:23,391 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:23,392 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:23,392 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,392 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,392 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:23,392 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,392 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:23,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:23,392 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:23,492 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:23,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:23,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:23,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:23,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:23,496 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:23,496 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:23,497 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:23,497 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:23,497 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,497 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:23,498 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:23,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:23,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:23,498 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:23,499 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,499 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:23,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:23,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:23,499 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:23,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:23,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:23,500 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:23,500 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:23,500 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:23,500 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:23,500 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:23,500 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:23,500 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:23,500 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:23,500 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:23,500 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:23,500 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:23,500 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,500 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:23,500 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:23,501 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,501 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:23,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:23,502 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,502 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:23,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:23,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:23,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:23,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:23,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:23,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@620745c[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:23,503 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:23,503 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,504 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,504 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:23,504 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:23,504 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:23,504 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,504 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:23,505 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:23,505 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,505 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:23,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:23,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:23,505 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:23,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:23,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:23,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:23,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:23,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:23,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:23,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:23,507 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:23,508 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:23,508 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:23,508 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:23,508 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:23,509 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,509 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:23,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,509 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:23,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:23,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:23,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:23,509 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:23,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:23,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:23,510 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:23,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:23,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:23,512 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,512 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:23,513 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:23,513 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:23,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:23,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:23,513 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:23,513 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:23,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:23,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:23,513 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:23,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40934 deadline: 1680987383512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:23,513 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:23,513 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:23,513 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,513 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,513 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:23,514 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:23,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40978 deadline: 1680987383512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:23,514 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:23,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40950 deadline: 1680987383512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,519 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:23,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40936 deadline: 1680987383516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,595 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/97e35320631b4b2f92ecfcec0928f2b9 2023-04-08 20:55:23,613 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:23,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:23,617 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/eb0acd8a6b1548ea81aaa09f53c8bee0 2023-04-08 20:55:23,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:23,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:23,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:23,623 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:23,624 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:23,624 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:23,624 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:23,624 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,624 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:23,625 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:23,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:23,625 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:23,626 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,626 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:23,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:23,626 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:23,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:23,627 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:23,630 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:23,630 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:23,630 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:23,630 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:23,630 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:23,634 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:23,648 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:23,648 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:23,649 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:23,651 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:23,651 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:23,651 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:23,651 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:23,651 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,651 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:23,651 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:23,652 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,652 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:23,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:23,653 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,653 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:23,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:23,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:23,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:23,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:23,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:23,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d752bd4[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:23,657 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:23,657 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,658 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,658 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:23,658 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,659 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:23,659 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:23,659 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:23,659 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,659 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:23,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:23,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:23,660 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:23,660 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:23,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:23,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:23,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:23,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:23,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:23,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:23,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:23,664 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:23,664 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:23,664 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:23,664 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:23,664 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:23,667 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/e88d2e0c3f154e189011ab9d2e4c0314 2023-04-08 20:55:23,675 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,675 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:23,675 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:23,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:23,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:23,676 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/97e35320631b4b2f92ecfcec0928f2b9 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/97e35320631b4b2f92ecfcec0928f2b9 2023-04-08 20:55:23,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:23,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:23,677 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:23,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:23,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:23,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:23,680 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:23,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,685 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/97e35320631b4b2f92ecfcec0928f2b9, entries=200, sequenceid=240, filesize=14.1 K 2023-04-08 20:55:23,686 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:23,686 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,686 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:23,686 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:23,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:23,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:23,686 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:23,686 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,686 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:23,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:23,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:23,686 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:23,686 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:23,686 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,687 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,687 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:23,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:23,687 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/eb0acd8a6b1548ea81aaa09f53c8bee0 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/eb0acd8a6b1548ea81aaa09f53c8bee0 2023-04-08 20:55:23,696 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/eb0acd8a6b1548ea81aaa09f53c8bee0, entries=150, sequenceid=240, filesize=11.8 K 2023-04-08 20:55:23,699 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/e88d2e0c3f154e189011ab9d2e4c0314 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/e88d2e0c3f154e189011ab9d2e4c0314 2023-04-08 20:55:23,710 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/e88d2e0c3f154e189011ab9d2e4c0314, entries=150, sequenceid=240, filesize=11.8 K 2023-04-08 20:55:23,712 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 7f864e4ec7cff383a8a9db2371cff040 in 548ms, sequenceid=240, compaction requested=true 2023-04-08 20:55:23,712 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:23,713 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:23,713 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:55:23,713 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:55:23,713 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:23,713 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:23,714 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38738 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:55:23,714 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): 7f864e4ec7cff383a8a9db2371cff040/A is initiating minor compaction (all files) 2023-04-08 20:55:23,715 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f864e4ec7cff383a8a9db2371cff040/A in TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:23,715 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/78779c6aea004a1e88fd4bfe778e3e47, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/5f330c06c12045029afddaab062b12c6, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/97e35320631b4b2f92ecfcec0928f2b9] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp, totalSize=37.8 K 2023-04-08 20:55:23,714 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36348 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:55:23,715 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 7f864e4ec7cff383a8a9db2371cff040/B is initiating minor compaction (all files) 2023-04-08 20:55:23,715 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f864e4ec7cff383a8a9db2371cff040/B in TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:23,715 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/f1e853fdff714a898e3b9ecddd7d4a0e, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/27557d97ffa4483c87ceedc332910c67, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/eb0acd8a6b1548ea81aaa09f53c8bee0] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp, totalSize=35.5 K 2023-04-08 20:55:23,715 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 78779c6aea004a1e88fd4bfe778e3e47, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1680987320711 2023-04-08 20:55:23,716 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting f1e853fdff714a898e3b9ecddd7d4a0e, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1680987320711 2023-04-08 20:55:23,724 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 27557d97ffa4483c87ceedc332910c67, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=225, earliestPutTs=1680987321886 2023-04-08 20:55:23,725 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 5f330c06c12045029afddaab062b12c6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=225, earliestPutTs=1680987321886 2023-04-08 20:55:23,729 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 97e35320631b4b2f92ecfcec0928f2b9, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1680987323038 2023-04-08 20:55:23,731 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting eb0acd8a6b1548ea81aaa09f53c8bee0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1680987323038 2023-04-08 20:55:23,760 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7f864e4ec7cff383a8a9db2371cff040#A#compaction#30 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:23,762 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 7f864e4ec7cff383a8a9db2371cff040#B#compaction#31 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:23,781 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/a2e32179002f4e68a3675efef1937bda as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/a2e32179002f4e68a3675efef1937bda 2023-04-08 20:55:23,786 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:23,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:23,788 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/dcafe900a36549e786086ff43d70182f as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/dcafe900a36549e786086ff43d70182f 2023-04-08 20:55:23,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:23,799 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7f864e4ec7cff383a8a9db2371cff040/A of 7f864e4ec7cff383a8a9db2371cff040 into a2e32179002f4e68a3675efef1937bda(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:23,804 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:23,804 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040., storeName=7f864e4ec7cff383a8a9db2371cff040/A, priority=13, startTime=1680987323713; duration=0sec 2023-04-08 20:55:23,801 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7f864e4ec7cff383a8a9db2371cff040/B of 7f864e4ec7cff383a8a9db2371cff040 into dcafe900a36549e786086ff43d70182f(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:23,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:23,805 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:23,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:23,805 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:55:23,805 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:23,805 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:23,805 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:23,805 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:23,805 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:23,805 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040., storeName=7f864e4ec7cff383a8a9db2371cff040/B, priority=13, startTime=1680987323713; duration=0sec 2023-04-08 20:55:23,806 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,806 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:23,806 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:23,807 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36348 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:55:23,807 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:23,807 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:23,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:23,807 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): 7f864e4ec7cff383a8a9db2371cff040/C is initiating minor compaction (all files) 2023-04-08 20:55:23,807 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f864e4ec7cff383a8a9db2371cff040/C in TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:23,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:23,807 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/4cd9ff4a596a445cbc0f3e8f28a93c2c, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/3229064441c64a68b104f585dbfe3fc7, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/e88d2e0c3f154e189011ab9d2e4c0314] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp, totalSize=35.5 K 2023-04-08 20:55:23,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,807 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,808 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:23,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:23,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:23,809 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 4cd9ff4a596a445cbc0f3e8f28a93c2c, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1680987320711 2023-04-08 20:55:23,809 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:23,809 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 3229064441c64a68b104f585dbfe3fc7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=225, earliestPutTs=1680987321886 2023-04-08 20:55:23,810 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting e88d2e0c3f154e189011ab9d2e4c0314, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1680987323038 2023-04-08 20:55:23,810 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:23,810 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:23,810 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:23,810 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:23,810 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:23,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:23,817 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7f864e4ec7cff383a8a9db2371cff040 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-08 20:55:23,817 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=A 2023-04-08 20:55:23,817 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:23,818 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=B 2023-04-08 20:55:23,818 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:23,818 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:23,818 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:23,818 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=C 2023-04-08 20:55:23,818 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:23,818 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:23,818 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:23,818 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:23,818 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:23,818 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:23,818 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:23,818 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,818 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:23,819 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:23,820 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:23,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:23,820 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:23,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:23,821 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,821 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:23,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:23,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:23,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:23,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39f3ba59[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:23,828 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:23,828 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,834 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,834 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:23,834 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:23,834 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:23,834 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,835 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:23,836 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:23,836 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:23,836 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:23,836 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:23,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:23,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:23,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:23,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:23,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:23,839 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7f864e4ec7cff383a8a9db2371cff040#C#compaction#32 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:23,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:23,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:23,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:23,842 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:23,842 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:23,842 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:23,842 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:23,842 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:23,844 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:23,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40950 deadline: 1680987383836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,845 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:23,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40934 deadline: 1680987383837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,845 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:23,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:23,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40936 deadline: 1680987383838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40978 deadline: 1680987383842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,853 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,854 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:23,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:23,854 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:23,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,856 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,858 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:23,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:23,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:23,859 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:23,859 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:23,858 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:23,859 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:23,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:23,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:23,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:23,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:23,862 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,862 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:23,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:23,862 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:23,862 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:23,862 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,862 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,862 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,862 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:23,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:23,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:23,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:23,902 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=261 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/cf7a7d38606e40c8b11ede7d6b783084 2023-04-08 20:55:23,914 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/8cebf97792664b31994035e31a78bb75 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/8cebf97792664b31994035e31a78bb75 2023-04-08 20:55:23,928 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7f864e4ec7cff383a8a9db2371cff040/C of 7f864e4ec7cff383a8a9db2371cff040 into 8cebf97792664b31994035e31a78bb75(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:23,928 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:23,928 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040., storeName=7f864e4ec7cff383a8a9db2371cff040/C, priority=13, startTime=1680987323713; duration=0sec 2023-04-08 20:55:23,928 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:23,934 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=261 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/4ae78f827ced4a189678ec399d58949b 2023-04-08 20:55:23,946 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:23,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40950 deadline: 1680987383946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,947 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:23,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40934 deadline: 1680987383946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,956 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:23,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40936 deadline: 1680987383954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,959 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:23,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:23,961 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=261 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/6b698b2beb3f4e609d216e733a926c3d 2023-04-08 20:55:23,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:23,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:23,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:23,963 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:23,963 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:23,963 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:23,963 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:23,964 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,964 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:23,965 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:23,965 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:23,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:23,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:23,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:23,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:23,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:23,968 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:23,968 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/cf7a7d38606e40c8b11ede7d6b783084 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/cf7a7d38606e40c8b11ede7d6b783084 2023-04-08 20:55:23,968 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,968 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:23,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:23,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40978 deadline: 1680987383967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,970 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:23,970 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:23,971 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:23,971 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:23,971 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:23,972 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:23,972 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:23,972 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:23,972 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:23,972 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:23,972 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:23,972 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:23,972 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:23,972 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,972 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:23,972 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:23,974 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,974 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:23,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:23,975 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/cf7a7d38606e40c8b11ede7d6b783084, entries=200, sequenceid=261, filesize=14.1 K 2023-04-08 20:55:23,976 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/4ae78f827ced4a189678ec399d58949b as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/4ae78f827ced4a189678ec399d58949b 2023-04-08 20:55:23,977 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:23,977 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:23,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:23,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:23,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:23,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:23,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67a64acb[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:23,979 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:23,979 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,983 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:23,983 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:23,983 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:23,983 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:23,983 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/4ae78f827ced4a189678ec399d58949b, entries=150, sequenceid=261, filesize=11.8 K 2023-04-08 20:55:23,983 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,983 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:23,985 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:23,985 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,985 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/6b698b2beb3f4e609d216e733a926c3d as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/6b698b2beb3f4e609d216e733a926c3d 2023-04-08 20:55:23,985 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:23,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:23,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:23,985 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:23,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:23,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:23,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:23,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:23,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:23,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:23,994 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/6b698b2beb3f4e609d216e733a926c3d, entries=150, sequenceid=261, filesize=11.8 K 2023-04-08 20:55:23,996 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 7f864e4ec7cff383a8a9db2371cff040 in 178ms, sequenceid=261, compaction requested=false 2023-04-08 20:55:23,996 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:24,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:24,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:24,003 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:24,003 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:24,003 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:24,003 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:24,003 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:24,004 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,004 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,004 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:24,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:24,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:24,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:24,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:24,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:24,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:24,006 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:24,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:24,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,007 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:24,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:24,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,016 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,016 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:24,016 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:24,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:24,016 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:24,016 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:24,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:24,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:24,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:24,016 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:24,017 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:24,017 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:24,017 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,017 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,017 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,017 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:24,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:24,117 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:24,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:24,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:24,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:24,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:24,121 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:24,121 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:24,121 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:24,121 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:24,121 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,121 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:24,122 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:24,122 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:24,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:24,123 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,123 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:24,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:24,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:24,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:24,123 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:24,124 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:24,124 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:24,124 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:24,124 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:24,124 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:24,124 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:24,124 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:24,124 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:24,124 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:24,124 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7f864e4ec7cff383a8a9db2371cff040 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-08 20:55:24,125 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=A 2023-04-08 20:55:24,125 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:24,125 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=B 2023-04-08 20:55:24,125 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:24,125 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=C 2023-04-08 20:55:24,125 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:24,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:24,134 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:24,162 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:24,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40954 deadline: 1680987384157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,165 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/dde5c34729654cb9b3159cd0ec7ee8d8 2023-04-08 20:55:24,166 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:24,166 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:24,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40950 deadline: 1680987384162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40934 deadline: 1680987384162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,170 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:24,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40936 deadline: 1680987384167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,174 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:24,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40978 deadline: 1680987384171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,178 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/e6a173b6c74046cc8e8e1931bb1b7289 2023-04-08 20:55:24,191 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/855215913b0e4539b88f24f67382b21b 2023-04-08 20:55:24,197 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/dde5c34729654cb9b3159cd0ec7ee8d8 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/dde5c34729654cb9b3159cd0ec7ee8d8 2023-04-08 20:55:24,203 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/dde5c34729654cb9b3159cd0ec7ee8d8, entries=150, sequenceid=276, filesize=11.9 K 2023-04-08 20:55:24,204 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/e6a173b6c74046cc8e8e1931bb1b7289 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/e6a173b6c74046cc8e8e1931bb1b7289 2023-04-08 20:55:24,211 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/e6a173b6c74046cc8e8e1931bb1b7289, entries=150, sequenceid=276, filesize=11.9 K 2023-04-08 20:55:24,214 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/855215913b0e4539b88f24f67382b21b as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/855215913b0e4539b88f24f67382b21b 2023-04-08 20:55:24,224 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/855215913b0e4539b88f24f67382b21b, entries=150, sequenceid=276, filesize=11.9 K 2023-04-08 20:55:24,226 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=114.05 KB/116790 for 7f864e4ec7cff383a8a9db2371cff040 in 101ms, sequenceid=276, compaction requested=true 2023-04-08 20:55:24,226 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:24,226 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:24,226 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:24,226 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:24,226 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,226 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:24,226 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:24,227 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,227 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:24,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:24,227 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,227 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:24,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:24,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:24,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:24,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:24,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:24,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62a150f7[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:24,229 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:24,229 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,230 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,230 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,230 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:24,230 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:24,230 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:24,230 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:24,232 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,232 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:24,232 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:24,232 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:24,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:24,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:24,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:24,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:24,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:24,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:24,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:24,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:24,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:24,234 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:24,235 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:24,235 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:24,235 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:24,235 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:24,236 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,236 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:24,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:24,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,236 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:24,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:24,236 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:24,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:24,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:24,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:24,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:24,237 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:24,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:24,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,239 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,239 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:24,239 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:24,239 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:24,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:24,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:24,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:24,239 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:24,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:24,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:24,239 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,239 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:24,239 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,239 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,239 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:24,239 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:24,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:24,240 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:24,266 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7f864e4ec7cff383a8a9db2371cff040 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-08 20:55:24,266 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=A 2023-04-08 20:55:24,266 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:55:24,266 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=B 2023-04-08 20:55:24,267 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:55:24,267 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=C 2023-04-08 20:55:24,267 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:55:24,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:24,279 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=297 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/42d80e3d72d34d6eb929052fe4a1351e 2023-04-08 20:55:24,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:24,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40934 deadline: 1680987384280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:24,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40950 deadline: 1680987384281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,290 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:24,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40954 deadline: 1680987384288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,291 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:24,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40936 deadline: 1680987384289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,340 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:24,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:24,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:24,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:24,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:24,344 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:24,344 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:24,344 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:24,344 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:24,344 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,345 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:24,346 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,346 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:24,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:24,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:24,347 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,347 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:24,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:24,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:24,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:24,347 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:24,347 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:24,347 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:24,347 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:24,348 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:24,348 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:24,348 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:24,348 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:24,348 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:24,348 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:24,348 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:24,348 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:24,348 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:24,348 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:24,348 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,348 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:24,348 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:24,349 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,349 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:24,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:24,349 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:24,349 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:24,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:24,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:24,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:24,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:24,350 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:24,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@761eecfc[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:24,350 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,351 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,351 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:24,351 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:24,351 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,351 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:24,351 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:24,352 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,352 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:24,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:24,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:24,352 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:24,352 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:24,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:24,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:24,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:24,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:24,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:24,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:24,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:24,358 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:24,358 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:24,358 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:24,358 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:24,358 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:24,359 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,359 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:24,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:24,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,359 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:24,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:24,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:24,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:24,360 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:24,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:24,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:24,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,361 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:24,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:24,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,363 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,363 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:24,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:24,363 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:24,363 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:24,363 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:24,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:24,364 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:24,364 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:24,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:24,363 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:24,363 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,363 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:24,364 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,364 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:24,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:24,389 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:24,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40934 deadline: 1680987384389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,391 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:24,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40950 deadline: 1680987384389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,395 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:24,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40954 deadline: 1680987384392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,395 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:24,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40936 deadline: 1680987384392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,464 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:24,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:24,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:24,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:24,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:24,468 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:24,468 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:24,468 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:24,468 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:24,469 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,469 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:24,469 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,469 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:24,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:24,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:24,470 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,470 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:24,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:24,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:24,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:24,470 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:24,471 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:24,471 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:24,471 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:24,471 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:24,471 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:24,471 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:24,471 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:24,471 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:24,471 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:24,471 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:24,471 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:24,471 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:24,471 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:24,471 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,471 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:24,471 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:24,472 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,472 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:24,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:24,472 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,472 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:24,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:24,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:24,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:24,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:24,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:24,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@529bb114[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:24,473 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:24,473 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,474 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,474 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:24,474 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:24,474 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:24,474 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,474 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:24,475 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,475 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:24,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:24,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:24,475 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:24,475 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:24,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:24,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:24,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:24,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:24,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:24,477 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:24,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40978 deadline: 1680987384476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:24,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:24,479 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:24,479 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:24,479 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:24,479 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:24,479 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:24,480 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,480 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,480 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:24,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:24,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:24,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,480 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:24,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:24,481 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:24,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:24,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:24,483 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,483 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:24,483 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:24,483 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:24,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:24,483 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:24,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:24,483 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,483 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:24,483 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:24,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:24,483 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:24,483 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,483 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:24,483 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:24,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:24,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:24,583 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:24,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:24,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:24,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:24,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:24,587 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:24,587 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:24,587 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:24,587 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:24,587 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,587 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:24,588 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:24,588 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:24,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:24,589 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,589 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:24,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:24,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:24,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:24,589 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:24,589 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:24,589 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:24,590 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:24,590 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:24,590 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:24,590 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:24,590 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:24,590 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:24,590 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:24,590 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:24,590 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:24,590 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:24,590 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:24,590 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,590 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:24,590 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:24,591 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,591 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:24,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:24,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:24,592 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,592 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:24,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:24,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:24,593 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:24,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40934 deadline: 1680987384591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:24,593 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:24,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40950 deadline: 1680987384592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:24,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73e499e[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:24,594 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:24,594 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,594 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,595 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:24,595 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:24,595 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:24,595 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,595 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:24,596 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:24,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:24,596 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:24,596 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:24,596 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:24,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:24,598 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:24,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40954 deadline: 1680987384596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,599 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:24,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40936 deadline: 1680987384598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:24,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:24,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:24,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:24,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:24,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:24,602 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:24,602 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:24,602 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:24,602 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:24,602 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:24,608 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,608 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,608 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:24,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:24,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:24,608 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:24,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:24,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:24,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:24,610 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:24,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:24,616 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,616 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:24,616 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:24,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:24,616 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:24,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:24,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:24,616 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:24,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:24,616 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:24,617 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,617 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:24,617 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:24,617 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,617 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:24,617 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:24,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:24,702 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=297 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/120080b2476841b8bec275b690eed642 2023-04-08 20:55:24,717 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:24,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:24,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:24,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:24,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:24,723 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:24,724 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:24,724 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:24,724 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:24,724 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,724 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:24,725 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,726 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,726 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:24,727 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:24,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:24,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:24,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:24,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:24,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:24,728 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:24,728 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:24,728 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:24,728 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:24,728 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:24,728 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:24,734 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:24,734 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:24,734 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:24,734 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:24,734 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:24,734 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:24,734 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:24,734 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:24,734 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,742 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:24,742 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:24,744 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:24,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:24,744 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=297 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/28fac68d77664a3f9f6bfbede1d7a99f 2023-04-08 20:55:24,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:24,744 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,745 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,745 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:24,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:24,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:24,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:24,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:24,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ff09766[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:24,747 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:24,747 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,758 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/42d80e3d72d34d6eb929052fe4a1351e as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/42d80e3d72d34d6eb929052fe4a1351e 2023-04-08 20:55:24,758 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,759 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:24,759 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:24,759 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:24,759 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,760 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:24,760 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,760 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:24,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:24,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:24,760 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:24,761 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:24,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:24,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:24,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:24,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:24,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:24,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:24,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:24,765 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:24,765 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:24,765 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:24,766 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:24,766 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:24,767 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/42d80e3d72d34d6eb929052fe4a1351e, entries=200, sequenceid=297, filesize=14.3 K 2023-04-08 20:55:24,767 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,767 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:24,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,767 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:24,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:24,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:24,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:24,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:24,768 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:24,769 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/120080b2476841b8bec275b690eed642 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/120080b2476841b8bec275b690eed642 2023-04-08 20:55:24,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:24,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:24,769 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:24,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:24,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,771 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,771 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:24,771 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:24,771 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,771 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:24,772 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,772 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:24,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:24,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:24,772 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:24,772 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:24,772 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:24,772 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:24,772 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:24,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:24,772 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:24,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:24,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:24,775 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/120080b2476841b8bec275b690eed642, entries=150, sequenceid=297, filesize=11.9 K 2023-04-08 20:55:24,776 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/28fac68d77664a3f9f6bfbede1d7a99f as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/28fac68d77664a3f9f6bfbede1d7a99f 2023-04-08 20:55:24,781 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/28fac68d77664a3f9f6bfbede1d7a99f, entries=150, sequenceid=297, filesize=11.9 K 2023-04-08 20:55:24,783 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=60.38 KB/61830 for 7f864e4ec7cff383a8a9db2371cff040 in 517ms, sequenceid=297, compaction requested=true 2023-04-08 20:55:24,783 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:24,783 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:24,783 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 20:55:24,783 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 20:55:24,783 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:24,783 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:24,784 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48836 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 20:55:24,784 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 7f864e4ec7cff383a8a9db2371cff040/B is initiating minor compaction (all files) 2023-04-08 20:55:24,784 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 53666 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 20:55:24,785 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f864e4ec7cff383a8a9db2371cff040/B in TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:24,785 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): 7f864e4ec7cff383a8a9db2371cff040/A is initiating minor compaction (all files) 2023-04-08 20:55:24,785 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/dcafe900a36549e786086ff43d70182f, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/4ae78f827ced4a189678ec399d58949b, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/e6a173b6c74046cc8e8e1931bb1b7289, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/120080b2476841b8bec275b690eed642] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp, totalSize=47.7 K 2023-04-08 20:55:24,785 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f864e4ec7cff383a8a9db2371cff040/A in TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:24,785 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/a2e32179002f4e68a3675efef1937bda, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/cf7a7d38606e40c8b11ede7d6b783084, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/dde5c34729654cb9b3159cd0ec7ee8d8, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/42d80e3d72d34d6eb929052fe4a1351e] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp, totalSize=52.4 K 2023-04-08 20:55:24,785 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting dcafe900a36549e786086ff43d70182f, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1680987323038 2023-04-08 20:55:24,785 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting a2e32179002f4e68a3675efef1937bda, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1680987323038 2023-04-08 20:55:24,786 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 4ae78f827ced4a189678ec399d58949b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1680987323202 2023-04-08 20:55:24,786 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting cf7a7d38606e40c8b11ede7d6b783084, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1680987323202 2023-04-08 20:55:24,786 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting e6a173b6c74046cc8e8e1931bb1b7289, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1680987323820 2023-04-08 20:55:24,787 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting dde5c34729654cb9b3159cd0ec7ee8d8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1680987323820 2023-04-08 20:55:24,787 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 120080b2476841b8bec275b690eed642, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1680987324156 2023-04-08 20:55:24,787 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 42d80e3d72d34d6eb929052fe4a1351e, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1680987324156 2023-04-08 20:55:24,800 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7f864e4ec7cff383a8a9db2371cff040#A#compaction#42 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:24,803 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 7f864e4ec7cff383a8a9db2371cff040#B#compaction#43 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:24,829 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/0a1467a2fd4a4382b279f613ef17115d as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/0a1467a2fd4a4382b279f613ef17115d 2023-04-08 20:55:24,831 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/68e6e04f5f314640bf21133ede450022 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/68e6e04f5f314640bf21133ede450022 2023-04-08 20:55:24,838 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7f864e4ec7cff383a8a9db2371cff040/B of 7f864e4ec7cff383a8a9db2371cff040 into 0a1467a2fd4a4382b279f613ef17115d(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:24,838 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:24,838 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040., storeName=7f864e4ec7cff383a8a9db2371cff040/B, priority=12, startTime=1680987324783; duration=0sec 2023-04-08 20:55:24,839 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:24,839 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 20:55:24,840 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7f864e4ec7cff383a8a9db2371cff040/A of 7f864e4ec7cff383a8a9db2371cff040 into 68e6e04f5f314640bf21133ede450022(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:24,840 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:24,841 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040., storeName=7f864e4ec7cff383a8a9db2371cff040/A, priority=12, startTime=1680987324783; duration=0sec 2023-04-08 20:55:24,841 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:24,841 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48836 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 20:55:24,841 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 7f864e4ec7cff383a8a9db2371cff040/C is initiating minor compaction (all files) 2023-04-08 20:55:24,841 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f864e4ec7cff383a8a9db2371cff040/C in TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:24,841 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/8cebf97792664b31994035e31a78bb75, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/6b698b2beb3f4e609d216e733a926c3d, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/855215913b0e4539b88f24f67382b21b, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/28fac68d77664a3f9f6bfbede1d7a99f] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp, totalSize=47.7 K 2023-04-08 20:55:24,842 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 8cebf97792664b31994035e31a78bb75, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1680987323038 2023-04-08 20:55:24,843 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 6b698b2beb3f4e609d216e733a926c3d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1680987323202 2023-04-08 20:55:24,843 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 855215913b0e4539b88f24f67382b21b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1680987323820 2023-04-08 20:55:24,844 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 28fac68d77664a3f9f6bfbede1d7a99f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1680987324156 2023-04-08 20:55:24,856 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 7f864e4ec7cff383a8a9db2371cff040#C#compaction#44 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:24,868 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/dd0d0d2ebd80415689438c87f7fee5e1 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/dd0d0d2ebd80415689438c87f7fee5e1 2023-04-08 20:55:24,872 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:24,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:24,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:24,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:24,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:24,878 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:24,878 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:24,878 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:24,879 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:24,879 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,879 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:24,880 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,880 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:24,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:24,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:24,880 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,880 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:24,880 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7f864e4ec7cff383a8a9db2371cff040/C of 7f864e4ec7cff383a8a9db2371cff040 into dd0d0d2ebd80415689438c87f7fee5e1(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:24,880 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:24,881 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040., storeName=7f864e4ec7cff383a8a9db2371cff040/C, priority=12, startTime=1680987324783; duration=0sec 2023-04-08 20:55:24,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:24,881 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:24,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:24,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:24,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:24,881 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:24,881 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:24,882 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:24,882 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:24,882 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:24,882 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:24,890 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:24,890 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:24,890 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:24,890 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:24,891 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7f864e4ec7cff383a8a9db2371cff040 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-08 20:55:24,891 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=A 2023-04-08 20:55:24,891 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:24,891 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=B 2023-04-08 20:55:24,891 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:24,891 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=C 2023-04-08 20:55:24,891 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:24,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:24,900 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:24,905 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/7aecca3687dd4dd19d4fdb9342c59e4c 2023-04-08 20:55:24,927 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1680987324911, max timestamp=1680987324924Num uniques 859; 2023-04-08 20:55:24,927 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1680987324911, max timestamp=1680987324924Num uniques 859; 2023-04-08 20:55:24,927 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-08 20:55:24,927 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1680987324911, max timestamp=1680987324924Num uniques 859; 2023-04-08 20:55:24,927 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-08 20:55:24,927 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-08 20:55:24,946 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:24,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:24,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40934 deadline: 1680987384941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40954 deadline: 1680987384943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:24,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40936 deadline: 1680987384944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,947 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:24,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40950 deadline: 1680987384944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:24,982 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:24,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40978 deadline: 1680987384980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,048 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:25,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40954 deadline: 1680987385048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,049 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:25,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40934 deadline: 1680987385048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,049 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:25,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40936 deadline: 1680987385048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,049 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:25,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40950 deadline: 1680987385049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,252 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:25,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40936 deadline: 1680987385251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,253 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:25,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40950 deadline: 1680987385251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,257 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:25,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40934 deadline: 1680987385253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,257 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:25,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40954 deadline: 1680987385254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,341 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/48df75b63a8140b8883823d994ea2172 2023-04-08 20:55:25,355 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/654353f6dded40aabd489a2289fb8553 2023-04-08 20:55:25,361 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/7aecca3687dd4dd19d4fdb9342c59e4c as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/7aecca3687dd4dd19d4fdb9342c59e4c 2023-04-08 20:55:25,367 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/7aecca3687dd4dd19d4fdb9342c59e4c, entries=150, sequenceid=312, filesize=11.9 K 2023-04-08 20:55:25,368 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/48df75b63a8140b8883823d994ea2172 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/48df75b63a8140b8883823d994ea2172 2023-04-08 20:55:25,374 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/48df75b63a8140b8883823d994ea2172, entries=150, sequenceid=312, filesize=11.9 K 2023-04-08 20:55:25,375 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/654353f6dded40aabd489a2289fb8553 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/654353f6dded40aabd489a2289fb8553 2023-04-08 20:55:25,380 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/654353f6dded40aabd489a2289fb8553, entries=150, sequenceid=312, filesize=11.9 K 2023-04-08 20:55:25,381 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=127.47 KB/130530 for 7f864e4ec7cff383a8a9db2371cff040 in 491ms, sequenceid=312, compaction requested=false 2023-04-08 20:55:25,381 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:25,381 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:25,382 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:25,382 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:25,382 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,382 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:25,382 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:25,386 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,386 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:25,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:25,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:25,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:25,386 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:25,386 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:25,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:25,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:25,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:25,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:25,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e08ef68[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:25,387 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:25,387 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:25,388 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:25,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:25,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:25,388 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,388 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:25,388 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:25,388 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:25,388 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:25,389 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,389 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:25,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:25,390 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:25,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:25,390 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:25,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:25,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:25,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:25,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:25,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:25,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:25,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:25,392 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:25,392 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:25,392 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:25,392 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:25,392 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:25,393 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:25,393 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:25,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:25,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:25,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:25,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:25,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:25,393 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:25,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:25,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:25,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:25,394 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:25,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:25,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:25,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:25,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,395 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:25,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:25,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,397 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:25,397 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:25,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:25,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:25,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:25,397 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:25,397 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,397 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:25,397 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:25,397 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,397 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:25,397 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:25,397 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:25,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:25,397 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:25,397 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:25,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:25,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:25,497 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:25,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:25,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:25,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:25,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:25,502 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:25,502 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:25,502 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:25,502 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:25,502 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:25,502 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:25,503 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:25,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:25,503 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:25,503 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,503 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:25,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:25,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:25,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:25,504 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:25,504 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:25,504 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:25,504 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:25,504 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:25,504 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:25,505 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:25,505 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:25,505 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:25,505 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:25,505 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7f864e4ec7cff383a8a9db2371cff040 3/3 column families, dataSize=127.47 KB heapSize=351 KB 2023-04-08 20:55:25,505 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=A 2023-04-08 20:55:25,505 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:55:25,505 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=B 2023-04-08 20:55:25,505 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:55:25,505 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=C 2023-04-08 20:55:25,506 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:55:25,521 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=345 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/460c334fb52f477090c13d81bc9d58bc 2023-04-08 20:55:25,545 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=345 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/3986a8f8d18345a4bab03eb22e7ac3ee 2023-04-08 20:55:25,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:25,557 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:25,571 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:25,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40936 deadline: 1680987385567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,574 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:25,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40950 deadline: 1680987385570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:25,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40954 deadline: 1680987385572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:25,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40934 deadline: 1680987385572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:25,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40936 deadline: 1680987385673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,677 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:25,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40950 deadline: 1680987385676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:25,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40954 deadline: 1680987385676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:25,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987385676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:25,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987385877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,879 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:25,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40954 deadline: 1680987385879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,879 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:25,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40950 deadline: 1680987385879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,882 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:25,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987385880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,964 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=345 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/2cfda02a78b44732affba35ca36a3b2e 2023-04-08 20:55:25,971 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/460c334fb52f477090c13d81bc9d58bc as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/460c334fb52f477090c13d81bc9d58bc 2023-04-08 20:55:25,976 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/460c334fb52f477090c13d81bc9d58bc, entries=150, sequenceid=345, filesize=11.9 K 2023-04-08 20:55:25,977 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/3986a8f8d18345a4bab03eb22e7ac3ee as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/3986a8f8d18345a4bab03eb22e7ac3ee 2023-04-08 20:55:25,983 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/3986a8f8d18345a4bab03eb22e7ac3ee, entries=150, sequenceid=345, filesize=11.9 K 2023-04-08 20:55:25,984 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/2cfda02a78b44732affba35ca36a3b2e as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/2cfda02a78b44732affba35ca36a3b2e 2023-04-08 20:55:25,991 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/2cfda02a78b44732affba35ca36a3b2e, entries=150, sequenceid=345, filesize=11.9 K 2023-04-08 20:55:25,992 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~350.95 KB/359376, currentSize=60.38 KB/61830 for 7f864e4ec7cff383a8a9db2371cff040 in 487ms, sequenceid=345, compaction requested=true 2023-04-08 20:55:25,992 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:25,992 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:25,992 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:25,992 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:25,992 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,993 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:25,994 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:25,995 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,995 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:25,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:25,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:25,995 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:25,995 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:25,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:25,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:25,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:25,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:25,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:25,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6114f0ac[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:25,996 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:25,996 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:25,997 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:25,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:25,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:25,997 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:25,997 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:25,997 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,997 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:25,997 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:25,998 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,998 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:25,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:25,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:25,998 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:25,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:25,999 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:25,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:26,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:26,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:26,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:26,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:26,001 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7f864e4ec7cff383a8a9db2371cff040 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-08 20:55:26,001 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=A 2023-04-08 20:55:26,001 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:26,001 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=B 2023-04-08 20:55:26,001 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:26,002 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=C 2023-04-08 20:55:26,002 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:26,003 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:26,003 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:26,003 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:26,003 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:26,003 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:26,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:26,005 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:26,005 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:26,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:26,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:26,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:26,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:26,005 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:26,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:26,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:26,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:26,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:26,006 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:26,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:26,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:26,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:26,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:26,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:26,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,015 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,022 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,022 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:26,022 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:26,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:26,023 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:26,023 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:26,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:26,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:26,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:26,023 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:26,023 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,023 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:26,023 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=358 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/a3c884af255542d892c1b831c04e3780 2023-04-08 20:55:26,023 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:26,023 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:26,023 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:26,023 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:26,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:26,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:26,074 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:26,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40978 deadline: 1680987386073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,123 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:26,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:26,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:26,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:26,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:26,127 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:26,127 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:26,127 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:26,127 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:26,127 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:26,128 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:26,128 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:26,128 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:26,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:26,129 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,129 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:26,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:26,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:26,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:26,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:26,129 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:26,129 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:26,129 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:26,130 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:26,130 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:26,130 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:26,130 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:26,130 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:26,130 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:26,130 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:26,130 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:26,130 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:26,130 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:26,130 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:26,130 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,130 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:26,130 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:26,143 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,143 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:26,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:26,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:26,144 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:26,144 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:26,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:26,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:26,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:26,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:26,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:26,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@141327a9[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:26,145 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:26,145 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:26,145 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:26,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:26,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:26,145 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,145 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:26,145 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:26,146 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:26,146 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:26,147 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:26,147 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,147 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:26,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:26,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:26,147 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:26,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:26,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:26,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:26,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:26,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:26,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:26,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:26,149 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:26,149 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:26,149 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:26,149 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:26,149 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:26,150 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:26,150 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:26,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:26,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:26,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:26,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:26,150 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:26,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:26,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:26,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:26,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:26,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:26,150 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:26,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:26,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:26,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:26,151 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:26,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,153 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,153 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:26,153 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:26,153 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:26,153 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:26,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:26,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:26,153 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:26,153 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:26,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:26,153 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:26,153 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:26,153 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:26,153 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:26,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:26,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:26,154 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:26,177 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:26,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40978 deadline: 1680987386176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,180 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:26,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987386179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,183 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:26,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40954 deadline: 1680987386181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,183 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:26,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40950 deadline: 1680987386182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:26,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987386185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,254 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:26,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:26,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:26,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:26,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:26,259 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:26,259 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:26,259 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:26,259 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:26,259 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:26,259 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:26,260 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:26,260 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:26,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:26,260 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,261 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:26,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:26,261 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:26,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:26,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:26,262 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:26,262 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:26,262 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:26,262 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:26,262 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:26,262 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:26,262 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:26,262 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:26,263 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:26,263 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:26,263 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:26,263 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:26,263 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:26,263 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:26,263 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,263 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:26,263 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:26,264 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,264 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:26,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:26,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:26,264 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:26,264 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:26,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:26,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:26,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:26,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:26,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:26,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fb3d9ea[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:26,266 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:26,266 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:26,267 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,268 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:26,270 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:26,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:26,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:26,270 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:26,270 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:26,270 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:26,271 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,271 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:26,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:26,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:26,271 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:26,271 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:26,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:26,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:26,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:26,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:26,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:26,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:26,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:26,277 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:26,277 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:26,277 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:26,277 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:26,277 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:26,283 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:26,283 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:26,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:26,283 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:26,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:26,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:26,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:26,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:26,283 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:26,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:26,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:26,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:26,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:26,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:26,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:26,284 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:26,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:26,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,286 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,286 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:26,286 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:26,286 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:26,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:26,286 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:26,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:26,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:26,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:26,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:26,286 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:26,287 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,287 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:26,287 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:26,287 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:26,287 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:26,287 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:26,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:26,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:26,381 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:26,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40978 deadline: 1680987386380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,387 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:26,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:26,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:26,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:26,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:26,390 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:26,390 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:26,390 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:26,390 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:26,391 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:26,391 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:26,391 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,391 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:26,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:26,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:26,392 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,392 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:26,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:26,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:26,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:26,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:26,392 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:26,394 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:26,394 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:26,394 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:26,394 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:26,394 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:26,394 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:26,394 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:26,394 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:26,395 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:26,395 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:26,395 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:26,395 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:26,395 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:26,395 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,395 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:26,395 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:26,396 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,396 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:26,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:26,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:26,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:26,397 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:26,397 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:26,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:26,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:26,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:26,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:26,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56690adb[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:26,398 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:26,398 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:26,399 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:26,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:26,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:26,399 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:26,399 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:26,399 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:26,399 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,399 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:26,402 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:26,402 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:26,402 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:26,402 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:26,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:26,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:26,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:26,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:26,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:26,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:26,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:26,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:26,408 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:26,408 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:26,408 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:26,408 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:26,408 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:26,416 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:26,416 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:26,416 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:26,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:26,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:26,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:26,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:26,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:26,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:26,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:26,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:26,417 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:26,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:26,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:26,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:26,419 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:26,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:26,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,424 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,424 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:26,424 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:26,424 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:26,424 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:26,425 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,425 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:26,425 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:26,425 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:26,425 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:26,425 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:26,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:26,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:26,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:26,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:26,425 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:26,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:26,425 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:26,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:26,443 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=358 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/1daca8843c5342f594bd679aa1f47288 2023-04-08 20:55:26,478 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=358 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/863acd97276a4d79aaa602cc8fe42544 2023-04-08 20:55:26,486 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/a3c884af255542d892c1b831c04e3780 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/a3c884af255542d892c1b831c04e3780 2023-04-08 20:55:26,491 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/a3c884af255542d892c1b831c04e3780, entries=150, sequenceid=358, filesize=11.9 K 2023-04-08 20:55:26,492 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/1daca8843c5342f594bd679aa1f47288 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/1daca8843c5342f594bd679aa1f47288 2023-04-08 20:55:26,517 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/1daca8843c5342f594bd679aa1f47288, entries=150, sequenceid=358, filesize=11.9 K 2023-04-08 20:55:26,519 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/863acd97276a4d79aaa602cc8fe42544 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/863acd97276a4d79aaa602cc8fe42544 2023-04-08 20:55:26,526 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:26,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:26,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:26,528 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/863acd97276a4d79aaa602cc8fe42544, entries=150, sequenceid=358, filesize=11.9 K 2023-04-08 20:55:26,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:26,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:26,529 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:26,529 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:26,529 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:26,529 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:26,530 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 7f864e4ec7cff383a8a9db2371cff040 in 529ms, sequenceid=358, compaction requested=true 2023-04-08 20:55:26,530 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:26,530 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:26,530 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:26,530 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:26,530 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 20:55:26,530 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:26,530 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 20:55:26,530 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:26,531 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:26,531 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:26,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:26,531 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,531 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:26,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:26,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:26,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:26,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:26,533 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:26,533 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49222 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 20:55:26,533 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): 7f864e4ec7cff383a8a9db2371cff040/A is initiating minor compaction (all files) 2023-04-08 20:55:26,533 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f864e4ec7cff383a8a9db2371cff040/A in TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:26,533 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/68e6e04f5f314640bf21133ede450022, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/7aecca3687dd4dd19d4fdb9342c59e4c, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/460c334fb52f477090c13d81bc9d58bc, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/a3c884af255542d892c1b831c04e3780] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp, totalSize=48.1 K 2023-04-08 20:55:26,538 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:26,538 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:26,538 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:26,538 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:26,538 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:26,538 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:26,538 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:26,538 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:26,538 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:26,539 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7f864e4ec7cff383a8a9db2371cff040 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-08 20:55:26,539 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=A 2023-04-08 20:55:26,539 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 68e6e04f5f314640bf21133ede450022, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1680987324156 2023-04-08 20:55:26,539 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:26,539 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49222 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 20:55:26,539 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=B 2023-04-08 20:55:26,539 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 7f864e4ec7cff383a8a9db2371cff040/B is initiating minor compaction (all files) 2023-04-08 20:55:26,539 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:26,539 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f864e4ec7cff383a8a9db2371cff040/B in TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:26,539 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=C 2023-04-08 20:55:26,539 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/0a1467a2fd4a4382b279f613ef17115d, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/48df75b63a8140b8883823d994ea2172, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/3986a8f8d18345a4bab03eb22e7ac3ee, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/1daca8843c5342f594bd679aa1f47288] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp, totalSize=48.1 K 2023-04-08 20:55:26,540 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:26,539 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 7aecca3687dd4dd19d4fdb9342c59e4c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1680987324273 2023-04-08 20:55:26,540 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 0a1467a2fd4a4382b279f613ef17115d, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1680987324156 2023-04-08 20:55:26,540 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 48df75b63a8140b8883823d994ea2172, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1680987324273 2023-04-08 20:55:26,541 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 460c334fb52f477090c13d81bc9d58bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1680987324939 2023-04-08 20:55:26,541 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 3986a8f8d18345a4bab03eb22e7ac3ee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1680987324939 2023-04-08 20:55:26,543 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 1daca8843c5342f594bd679aa1f47288, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1680987325565 2023-04-08 20:55:26,543 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting a3c884af255542d892c1b831c04e3780, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1680987325565 2023-04-08 20:55:26,563 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 7f864e4ec7cff383a8a9db2371cff040#B#compaction#55 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:26,569 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7f864e4ec7cff383a8a9db2371cff040#A#compaction#56 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:26,592 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/9c3ab4a5ad5d4ef4ac219995a7f34c5d as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/9c3ab4a5ad5d4ef4ac219995a7f34c5d 2023-04-08 20:55:26,599 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7f864e4ec7cff383a8a9db2371cff040/B of 7f864e4ec7cff383a8a9db2371cff040 into 9c3ab4a5ad5d4ef4ac219995a7f34c5d(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:26,599 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:26,599 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040., storeName=7f864e4ec7cff383a8a9db2371cff040/B, priority=12, startTime=1680987326530; duration=0sec 2023-04-08 20:55:26,599 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:26,599 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 20:55:26,604 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49222 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 20:55:26,604 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 7f864e4ec7cff383a8a9db2371cff040/C is initiating minor compaction (all files) 2023-04-08 20:55:26,604 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f864e4ec7cff383a8a9db2371cff040/C in TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:26,604 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/6cdb90273ec940e3b758cac50556c382 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/6cdb90273ec940e3b758cac50556c382 2023-04-08 20:55:26,604 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/dd0d0d2ebd80415689438c87f7fee5e1, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/654353f6dded40aabd489a2289fb8553, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/2cfda02a78b44732affba35ca36a3b2e, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/863acd97276a4d79aaa602cc8fe42544] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp, totalSize=48.1 K 2023-04-08 20:55:26,605 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting dd0d0d2ebd80415689438c87f7fee5e1, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1680987324156 2023-04-08 20:55:26,605 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 654353f6dded40aabd489a2289fb8553, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1680987324273 2023-04-08 20:55:26,606 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 2cfda02a78b44732affba35ca36a3b2e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1680987324939 2023-04-08 20:55:26,607 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 863acd97276a4d79aaa602cc8fe42544, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1680987325565 2023-04-08 20:55:26,612 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7f864e4ec7cff383a8a9db2371cff040/A of 7f864e4ec7cff383a8a9db2371cff040 into 6cdb90273ec940e3b758cac50556c382(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:26,612 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:26,612 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040., storeName=7f864e4ec7cff383a8a9db2371cff040/A, priority=12, startTime=1680987326530; duration=0sec 2023-04-08 20:55:26,612 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:26,620 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 7f864e4ec7cff383a8a9db2371cff040#C#compaction#57 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:26,630 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/d1ede29d5b8e4957a4d6bf8f8d2f603e as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/d1ede29d5b8e4957a4d6bf8f8d2f603e 2023-04-08 20:55:26,636 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7f864e4ec7cff383a8a9db2371cff040/C of 7f864e4ec7cff383a8a9db2371cff040 into d1ede29d5b8e4957a4d6bf8f8d2f603e(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:26,636 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:26,636 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040., storeName=7f864e4ec7cff383a8a9db2371cff040/C, priority=12, startTime=1680987326530; duration=0sec 2023-04-08 20:55:26,636 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:26,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:26,685 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:26,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:26,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987386693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,698 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:26,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40950 deadline: 1680987386693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,700 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:26,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40954 deadline: 1680987386696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,700 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:26,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:26,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987386696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40978 deadline: 1680987386697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,799 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:26,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987386798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,802 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:26,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40954 deadline: 1680987386801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:26,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40978 deadline: 1680987386802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:26,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987386803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,805 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:26,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40950 deadline: 1680987386805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:26,981 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=377 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/7521ecd8050e4de980d763528924a5f2 2023-04-08 20:55:27,001 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:27,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987387000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:27,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:27,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40954 deadline: 1680987387004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:27,006 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:27,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987387004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:27,006 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:27,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40978 deadline: 1680987387005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:27,008 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:27,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40950 deadline: 1680987387007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:27,305 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:27,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987387303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:27,308 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:27,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40954 deadline: 1680987387306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:27,310 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:27,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40978 deadline: 1680987387307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:27,312 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:27,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987387310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:27,313 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:27,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40950 deadline: 1680987387310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:27,398 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=377 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/f92f86b93b474bee9f0a516a3cfd966e 2023-04-08 20:55:27,809 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:27,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987387808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:27,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:27,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40954 deadline: 1680987387812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:27,813 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:27,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40978 deadline: 1680987387812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:27,820 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:27,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40950 deadline: 1680987387819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:27,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:27,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987387821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:27,830 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=377 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/ba57cacb1d0c41aa81f3e7a64f2d08ba 2023-04-08 20:55:27,836 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/7521ecd8050e4de980d763528924a5f2 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/7521ecd8050e4de980d763528924a5f2 2023-04-08 20:55:27,841 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/7521ecd8050e4de980d763528924a5f2, entries=150, sequenceid=377, filesize=11.9 K 2023-04-08 20:55:27,842 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/f92f86b93b474bee9f0a516a3cfd966e as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/f92f86b93b474bee9f0a516a3cfd966e 2023-04-08 20:55:27,847 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/f92f86b93b474bee9f0a516a3cfd966e, entries=150, sequenceid=377, filesize=11.9 K 2023-04-08 20:55:27,849 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/ba57cacb1d0c41aa81f3e7a64f2d08ba as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/ba57cacb1d0c41aa81f3e7a64f2d08ba 2023-04-08 20:55:27,854 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/ba57cacb1d0c41aa81f3e7a64f2d08ba, entries=150, sequenceid=377, filesize=11.9 K 2023-04-08 20:55:27,855 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 7f864e4ec7cff383a8a9db2371cff040 in 1316ms, sequenceid=377, compaction requested=false 2023-04-08 20:55:27,855 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:27,855 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:27,855 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:27,855 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:27,855 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:27,857 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:27,857 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:27,858 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:27,858 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:27,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:27,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:27,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:27,858 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:27,858 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:27,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:27,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:27,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:27,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:27,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:27,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:27,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10be112a[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:27,859 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:27,859 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:27,860 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:27,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:27,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:27,860 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:27,860 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:27,860 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:27,860 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:27,860 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:27,861 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:27,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:27,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:27,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:27,861 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:27,861 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:27,861 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:27,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:27,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:27,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:27,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:27,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:27,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:27,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:27,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:27,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:27,864 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:27,864 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:27,864 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:27,864 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:27,864 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:27,866 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:27,866 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:27,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:27,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:27,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:27,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:27,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:27,866 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:27,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:27,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:27,867 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:27,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:27,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:27,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:27,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:27,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:27,868 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:27,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:27,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:27,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:27,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:27,870 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:27,870 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:27,870 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:27,870 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:27,870 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:27,870 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:27,870 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:27,870 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:27,870 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:27,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:27,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:27,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:27,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:27,870 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:27,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:27,871 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:27,871 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:27,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:27,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:27,971 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:27,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:27,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:27,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:27,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:27,975 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:27,975 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:27,975 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:27,975 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:27,976 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:27,976 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:27,977 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:27,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:27,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:27,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:27,977 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:27,978 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:27,978 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:27,978 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:27,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:27,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:27,978 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:27,979 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:27,979 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:27,979 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:27,979 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:27,979 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:27,979 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:27,979 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:27,979 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:27,979 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7f864e4ec7cff383a8a9db2371cff040 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-08 20:55:27,980 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=A 2023-04-08 20:55:27,980 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:27,980 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=B 2023-04-08 20:55:27,980 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:27,980 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=C 2023-04-08 20:55:27,980 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:28,416 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=393 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/3a8226dbbec14c518eb59cc8b0a8b377 2023-04-08 20:55:28,468 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=393 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/dbf60d47ac844db6a8b54ccd413ac49a 2023-04-08 20:55:28,483 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=393 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/6d654e05b62a43c4b644ad070b2e2f16 2023-04-08 20:55:28,496 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/3a8226dbbec14c518eb59cc8b0a8b377 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/3a8226dbbec14c518eb59cc8b0a8b377 2023-04-08 20:55:28,504 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/3a8226dbbec14c518eb59cc8b0a8b377, entries=150, sequenceid=393, filesize=11.9 K 2023-04-08 20:55:28,507 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/dbf60d47ac844db6a8b54ccd413ac49a as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/dbf60d47ac844db6a8b54ccd413ac49a 2023-04-08 20:55:28,522 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/dbf60d47ac844db6a8b54ccd413ac49a, entries=150, sequenceid=393, filesize=11.9 K 2023-04-08 20:55:28,523 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/6d654e05b62a43c4b644ad070b2e2f16 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/6d654e05b62a43c4b644ad070b2e2f16 2023-04-08 20:55:28,530 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/6d654e05b62a43c4b644ad070b2e2f16, entries=150, sequenceid=393, filesize=11.9 K 2023-04-08 20:55:28,532 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for 7f864e4ec7cff383a8a9db2371cff040 in 553ms, sequenceid=393, compaction requested=true 2023-04-08 20:55:28,532 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:28,532 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:28,532 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:28,532 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:28,532 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,532 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:28,532 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:28,533 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,533 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:28,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:28,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:28,533 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:28,533 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:28,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:28,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:28,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:28,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:28,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d9d42fb[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:28,537 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:28,537 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:28,557 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:28,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:28,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:28,558 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:28,558 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:28,558 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:28,558 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,558 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:28,559 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,559 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:28,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:28,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:28,559 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:28,559 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:28,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:28,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:28,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:28,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:28,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:28,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:28,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:28,563 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:28,563 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:28,563 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:28,563 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:28,563 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:28,566 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:28,566 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:28,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:28,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:28,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:28,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:28,566 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:28,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:28,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:28,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:28,567 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:28,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:28,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:28,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:28,569 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:28,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,574 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:28,574 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:28,574 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:28,574 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:28,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:28,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:28,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:28,575 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:28,575 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:28,574 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:28,575 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:28,575 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,575 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:28,575 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:28,575 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:28,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:28,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:28,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:28,675 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:28,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:28,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:28,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:28,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:28,680 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:28,680 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:28,680 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:28,680 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:28,681 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:28,681 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:28,682 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:28,682 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:28,682 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,682 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:28,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:28,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:28,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:28,683 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:28,696 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:28,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:28,696 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:28,696 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:28,696 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:28,698 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:28,698 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:28,698 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:28,698 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:28,698 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:28,698 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:28,698 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:28,698 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:28,698 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,698 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:28,699 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:28,701 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,701 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:28,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:28,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:28,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:28,701 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:28,701 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:28,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:28,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:28,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:28,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:28,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c00a42b[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:28,703 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:28,703 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:28,705 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:28,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:28,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:28,705 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,705 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:28,705 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:28,705 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:28,705 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:28,706 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:28,706 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,706 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:28,706 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:28,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:28,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:28,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:28,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:28,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:28,708 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:28,708 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:28,708 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:28,708 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:28,708 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:28,709 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:28,709 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:28,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:28,709 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:28,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:28,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:28,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:28,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:28,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:28,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:28,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:28,710 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:28,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:28,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:28,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:28,712 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:28,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:28,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,713 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:28,713 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:28,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:28,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:28,713 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:28,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:28,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:28,714 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,714 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:28,714 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:28,714 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,714 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:28,714 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:28,714 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:28,714 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:28,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:28,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:28,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:28,714 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:28,814 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:28,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:28,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:28,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:28,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:28,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:28,819 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7f864e4ec7cff383a8a9db2371cff040 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-08 20:55:28,819 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:28,819 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:28,819 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:28,819 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:28,819 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=A 2023-04-08 20:55:28,819 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:28,819 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=B 2023-04-08 20:55:28,819 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:28,819 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:28,819 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=C 2023-04-08 20:55:28,819 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:28,819 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:28,820 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:28,820 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:28,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:28,820 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,820 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:28,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:28,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:28,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:28,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:28,821 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:28,822 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:28,822 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:28,822 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:28,822 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:28,822 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:28,822 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:28,822 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:28,822 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:28,822 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:28,822 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:28,822 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:28,822 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:28,822 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:28,822 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,822 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:28,822 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:28,823 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:28,824 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:28,824 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:28,824 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:28,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:28,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:28,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:28,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:28,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62a146a2[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:28,827 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:28,827 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:28,828 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:28,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:28,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:28,828 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:28,829 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:28,829 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:28,829 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,829 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:28,830 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,830 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:28,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:28,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:28,830 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:28,830 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:28,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:28,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:28,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:28,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:28,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:28,833 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:28,833 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:28,833 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:28,833 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:28,833 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:28,841 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:28,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:28,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:28,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:28,841 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:28,841 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680987328836, max timestamp=1680987328839Num uniques 859; 2023-04-08 20:55:28,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:28,841 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:28,841 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680987328836, max timestamp=1680987328839Num uniques 859; 2023-04-08 20:55:28,841 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680987328836, max timestamp=1680987328839Num uniques 859; 2023-04-08 20:55:28,841 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 20:55:28,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:28,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:28,841 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 20:55:28,841 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 20:55:28,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:28,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:28,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:28,842 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:28,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:28,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:28,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,844 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:28,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:28,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,850 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,850 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:28,850 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:28,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:28,850 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:28,850 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:28,850 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,851 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:28,851 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:28,851 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:28,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:28,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:28,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:28,850 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:28,851 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:28,851 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:28,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:28,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:28,860 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:28,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987388854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,860 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:28,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987388855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,860 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:28,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40954 deadline: 1680987388855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:28,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40950 deadline: 1680987388856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:28,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40978 deadline: 1680987388858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,951 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:28,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:28,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:28,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:28,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:28,957 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:28,957 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:28,957 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:28,957 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:28,957 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:28,957 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:28,958 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:28,958 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:28,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:28,959 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,959 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:28,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:28,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:28,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:28,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:28,960 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:28,965 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:28,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987388961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:28,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40954 deadline: 1680987388961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:28,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987388962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,968 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:28,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40978 deadline: 1680987388962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,968 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:28,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40950 deadline: 1680987388962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,979 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:28,979 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:28,979 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:28,979 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:28,979 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:28,981 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:28,981 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:28,981 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:28,981 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:28,982 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:28,982 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:28,982 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:28,982 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:28,982 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,982 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:28,982 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:28,984 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,984 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:28,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:28,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:28,985 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:28,985 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:28,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:28,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:28,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:28,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:28,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:28,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f62c9d[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:28,987 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:28,987 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:28,988 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:28,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:28,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:28,988 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:28,988 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:28,988 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:28,988 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,988 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:28,989 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,989 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:28,989 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:28,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:28,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:28,989 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:28,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:28,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:28,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:28,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:28,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:28,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:28,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:28,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:28,993 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:28,993 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:28,993 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:28,993 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:28,993 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:28,999 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:28,999 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:28,999 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:28,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:29,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:28,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,000 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:29,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:29,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:29,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:29,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,002 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:29,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,008 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:29,008 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:29,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:29,008 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:29,008 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,008 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,008 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:29,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:29,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:29,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:29,008 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,009 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,009 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,009 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,009 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:29,009 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:29,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:29,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:29,109 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:29,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:29,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:29,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:29,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:29,112 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:29,112 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:29,112 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:29,112 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:29,112 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,112 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,113 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:29,113 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:29,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:29,113 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,113 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:29,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:29,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,114 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:29,114 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:29,114 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:29,114 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:29,114 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:29,114 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:29,118 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:29,118 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:29,118 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:29,118 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:29,118 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:29,118 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:29,118 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:29,118 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:29,118 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,118 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:29,118 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:29,119 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,119 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:29,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:29,119 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,120 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:29,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:29,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:29,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:29,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:29,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@750b2ca2[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:29,121 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:29,121 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,121 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,122 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:29,122 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:29,122 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,122 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:29,122 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:29,122 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:29,122 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,123 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:29,123 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:29,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:29,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:29,125 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:29,125 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:29,125 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:29,125 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:29,125 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:29,126 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,126 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:29,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,126 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:29,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:29,127 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:29,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,128 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:29,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,141 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:29,141 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:29,141 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:29,141 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:29,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:29,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:29,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:29,141 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,141 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,141 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:29,141 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:29,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:29,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:29,141 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,142 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,141 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:29,142 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,168 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:29,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987389166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,170 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:29,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40950 deadline: 1680987389169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,173 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:29,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40954 deadline: 1680987389173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,173 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:29,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40978 deadline: 1680987389173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,174 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:29,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987389173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,230 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/952f047109d24c80803402df88ab8f85 2023-04-08 20:55:29,242 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:29,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:29,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:29,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:29,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:29,245 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:29,245 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:29,245 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:29,245 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:29,245 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,245 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,246 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:29,246 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:29,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:29,246 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,246 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:29,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:29,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,247 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:29,247 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:29,247 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:29,248 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:29,248 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:29,248 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:29,248 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:29,248 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:29,248 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:29,248 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:29,248 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:29,248 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:29,248 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:29,248 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:29,248 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,248 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:29,248 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:29,249 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,249 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:29,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:29,249 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,249 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:29,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:29,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:29,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:29,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:29,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c53efd4[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:29,251 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:29,251 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,251 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,252 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:29,252 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:29,252 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:29,252 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,252 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:29,252 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,252 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:29,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:29,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:29,252 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:29,253 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:29,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:29,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:29,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:29,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:29,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:29,254 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:29,255 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:29,255 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:29,255 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:29,255 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:29,255 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,255 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:29,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:29,255 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:29,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:29,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:29,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:29,256 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:29,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,258 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:29,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,261 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,261 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:29,261 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:29,261 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,261 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:29,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:29,262 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,262 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,262 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:29,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:29,262 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:29,262 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:29,262 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:29,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:29,262 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:29,362 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:29,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:29,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:29,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:29,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:29,365 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:29,365 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:29,365 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:29,365 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:29,366 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,366 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,367 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:29,367 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:29,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:29,367 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,367 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:29,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:29,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,368 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:29,368 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:29,368 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:29,368 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:29,368 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:29,368 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:29,368 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:29,368 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:29,368 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:29,368 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:29,368 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:29,368 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:29,368 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:29,368 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:29,368 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,368 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:29,368 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:29,369 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,369 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:29,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:29,369 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,369 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:29,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:29,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:29,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:29,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:29,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@642f3caa[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:29,370 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:29,370 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,371 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,371 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,371 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:29,373 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:29,373 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:29,373 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:29,373 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,373 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:29,374 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:29,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,374 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:29,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:29,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:29,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:29,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:29,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:29,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:29,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:29,376 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:29,376 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:29,376 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:29,376 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:29,376 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:29,377 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,377 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:29,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:29,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,377 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:29,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:29,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:29,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:29,377 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:29,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,378 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:29,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,380 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,380 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,380 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:29,380 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,380 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:29,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:29,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:29,380 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:29,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:29,380 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,380 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:29,380 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:29,380 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,380 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:29,380 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:29,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:29,472 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:29,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987389471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:29,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40950 deadline: 1680987389473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:29,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40954 deadline: 1680987389476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,479 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:29,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40978 deadline: 1680987389476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,479 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:29,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987389477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,480 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:29,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:29,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:29,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:29,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:29,483 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:29,483 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:29,483 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:29,483 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:29,483 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,483 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,484 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,484 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:29,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:29,484 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,484 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:29,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,485 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:29,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,485 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:29,485 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:29,485 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:29,486 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:29,486 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:29,486 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:29,486 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:29,486 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:29,486 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:29,486 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:29,486 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:29,486 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:29,486 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:29,486 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:29,486 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,486 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:29,486 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:29,487 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,487 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:29,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:29,487 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:29,488 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:29,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:29,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:29,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:29,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@578e6676[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:29,489 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:29,489 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,490 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,490 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:29,490 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:29,490 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:29,490 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,490 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:29,491 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,491 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:29,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:29,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:29,491 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:29,491 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:29,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:29,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:29,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:29,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:29,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:29,494 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:29,494 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:29,494 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:29,494 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:29,494 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:29,495 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,495 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:29,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:29,495 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:29,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:29,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:29,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:29,496 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:29,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,497 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:29,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,498 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,498 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:29,498 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:29,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:29,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:29,498 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,498 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:29,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:29,498 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:29,498 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:29,498 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:29,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:29,498 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,498 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,498 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,498 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:29,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:29,598 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:29,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:29,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:29,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:29,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:29,601 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:29,601 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:29,601 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:29,601 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:29,602 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,602 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,603 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,603 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:29,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:29,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:29,603 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,603 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:29,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:29,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,604 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:29,604 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:29,604 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:29,604 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:29,604 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:29,604 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:29,604 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:29,604 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:29,604 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:29,605 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:29,605 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:29,605 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:29,605 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:29,605 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:29,605 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,605 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:29,605 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:29,606 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,606 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:29,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:29,606 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,606 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:29,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:29,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:29,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:29,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:29,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b4c3e19[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:29,608 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:29,608 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,608 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,608 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:29,608 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:29,608 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,608 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:29,608 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:29,609 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,609 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:29,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:29,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:29,609 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:29,610 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:29,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:29,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:29,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:29,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:29,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:29,611 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:29,611 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:29,612 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:29,612 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:29,612 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:29,613 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:29,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:29,613 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,613 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:29,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:29,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:29,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:29,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,614 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:29,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,615 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:29,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,617 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,617 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,617 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,617 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:29,617 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:29,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:29,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:29,618 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,618 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,618 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,618 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:29,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:29,622 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:29,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:29,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:29,622 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:29,622 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:29,643 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/15f0c8b70cc941dfa19f73e848f801c4 2023-04-08 20:55:29,722 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:29,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:29,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:29,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:29,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:29,725 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:29,725 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:29,725 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:29,725 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:29,726 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,726 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,726 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,726 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:29,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:29,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:29,727 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,727 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:29,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:29,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,727 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:29,728 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:29,728 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:29,728 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:29,728 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:29,728 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:29,728 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:29,728 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:29,728 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:29,728 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:29,728 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:29,728 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:29,728 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:29,728 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:29,728 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,728 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:29,728 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:29,730 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,730 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:29,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:29,730 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,730 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:29,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:29,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:29,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:29,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:29,731 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:29,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@687243eb[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:29,731 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,731 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,732 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:29,732 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:29,732 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,732 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:29,732 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:29,732 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,732 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:29,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:29,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:29,732 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:29,732 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:29,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:29,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:29,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:29,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:29,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:29,734 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:29,734 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:29,734 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:29,734 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:29,734 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:29,735 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,735 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:29,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:29,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,735 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:29,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,735 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:29,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:29,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:29,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:29,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,736 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:29,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,738 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:29,738 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:29,738 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:29,738 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:29,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:29,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:29,738 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,738 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:29,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:29,739 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:29,738 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,739 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:29,739 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,739 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,739 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:29,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:29,839 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:29,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:29,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:29,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:29,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:29,842 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:29,842 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:29,842 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:29,842 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:29,842 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,842 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,843 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:29,843 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:29,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:29,843 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,843 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:29,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:29,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,844 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:29,844 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:29,844 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:29,844 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:29,844 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:29,844 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:29,844 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:29,844 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:29,844 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:29,844 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:29,844 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:29,844 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:29,844 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:29,845 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:29,845 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,845 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:29,845 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:29,846 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,846 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:29,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:29,846 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,846 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:29,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:29,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:29,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:29,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:29,847 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:29,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75a5f823[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:29,847 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,848 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,848 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:29,848 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:29,848 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:29,848 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,848 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:29,849 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,849 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:29,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:29,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:29,849 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:29,849 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:29,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:29,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:29,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:29,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:29,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:29,851 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:29,851 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:29,851 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:29,851 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:29,851 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:29,852 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,852 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:29,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:29,852 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:29,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:29,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:29,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:29,853 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:29,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,854 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:29,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,856 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:29,856 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:29,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:29,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:29,856 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:29,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:29,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:29,856 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:29,856 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,856 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,856 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,856 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,857 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:29,857 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:29,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:29,857 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:29,857 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,957 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:29,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:29,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:29,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:29,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:29,960 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:29,960 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:29,960 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:29,960 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:29,960 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,960 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,961 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:29,961 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:29,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:29,961 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,961 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:29,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:29,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,962 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:29,962 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:29,962 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:29,962 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:29,962 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:29,962 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:29,969 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:29,970 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:29,970 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:29,970 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:29,970 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:29,970 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:29,970 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:29,970 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:29,970 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,970 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:29,970 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:29,976 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,976 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:29,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:29,976 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,976 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:29,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:29,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:29,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:29,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:29,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@513c8d37[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:29,977 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:29,978 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,978 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:29,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987389975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,978 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:29,978 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40950 deadline: 1680987389977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,979 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:29,979 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:29,979 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:29,980 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,980 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:29,981 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:29,981 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,981 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:29,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40954 deadline: 1680987389981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:29,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:29,981 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:29,982 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:29,982 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:29,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987389981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:29,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:29,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:29,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:29,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:29,984 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:29,984 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:29,984 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:29,984 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:29,984 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:29,984 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:29,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40978 deadline: 1680987389983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,985 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,985 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:29,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:29,985 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:29,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:29,986 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:29,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:29,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:29,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:29,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,987 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:29,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:29,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,988 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:29,988 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:29,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:29,988 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,988 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:29,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:29,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:29,988 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:29,989 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:29,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:29,989 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:29,989 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:29,988 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:29,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:29,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:29,989 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,989 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:29,989 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:30,053 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/892a475f03044232a67b6a0608b0458f 2023-04-08 20:55:30,059 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/952f047109d24c80803402df88ab8f85 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/952f047109d24c80803402df88ab8f85 2023-04-08 20:55:30,063 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/952f047109d24c80803402df88ab8f85, entries=200, sequenceid=404, filesize=14.3 K 2023-04-08 20:55:30,064 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/15f0c8b70cc941dfa19f73e848f801c4 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/15f0c8b70cc941dfa19f73e848f801c4 2023-04-08 20:55:30,069 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/15f0c8b70cc941dfa19f73e848f801c4, entries=100, sequenceid=404, filesize=9.5 K 2023-04-08 20:55:30,070 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/892a475f03044232a67b6a0608b0458f as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/892a475f03044232a67b6a0608b0458f 2023-04-08 20:55:30,074 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/892a475f03044232a67b6a0608b0458f, entries=100, sequenceid=404, filesize=9.5 K 2023-04-08 20:55:30,074 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 7f864e4ec7cff383a8a9db2371cff040 in 1255ms, sequenceid=404, compaction requested=true 2023-04-08 20:55:30,075 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:30,075 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:30,075 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 20:55:30,075 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:30,075 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 20:55:30,075 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:30,076 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51794 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 20:55:30,076 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46914 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 20:55:30,076 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): 7f864e4ec7cff383a8a9db2371cff040/A is initiating minor compaction (all files) 2023-04-08 20:55:30,076 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 7f864e4ec7cff383a8a9db2371cff040/B is initiating minor compaction (all files) 2023-04-08 20:55:30,076 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f864e4ec7cff383a8a9db2371cff040/A in TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:30,076 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f864e4ec7cff383a8a9db2371cff040/B in TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:30,076 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/6cdb90273ec940e3b758cac50556c382, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/7521ecd8050e4de980d763528924a5f2, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/3a8226dbbec14c518eb59cc8b0a8b377, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/952f047109d24c80803402df88ab8f85] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp, totalSize=50.6 K 2023-04-08 20:55:30,076 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/9c3ab4a5ad5d4ef4ac219995a7f34c5d, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/f92f86b93b474bee9f0a516a3cfd966e, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/dbf60d47ac844db6a8b54ccd413ac49a, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/15f0c8b70cc941dfa19f73e848f801c4] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp, totalSize=45.8 K 2023-04-08 20:55:30,077 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 6cdb90273ec940e3b758cac50556c382, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1680987325565 2023-04-08 20:55:30,077 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 9c3ab4a5ad5d4ef4ac219995a7f34c5d, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1680987325565 2023-04-08 20:55:30,077 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 7521ecd8050e4de980d763528924a5f2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1680987326052 2023-04-08 20:55:30,077 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting f92f86b93b474bee9f0a516a3cfd966e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1680987326052 2023-04-08 20:55:30,078 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 3a8226dbbec14c518eb59cc8b0a8b377, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1680987326691 2023-04-08 20:55:30,078 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting dbf60d47ac844db6a8b54ccd413ac49a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1680987326691 2023-04-08 20:55:30,078 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 952f047109d24c80803402df88ab8f85, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1680987328815 2023-04-08 20:55:30,078 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 15f0c8b70cc941dfa19f73e848f801c4, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1680987328816 2023-04-08 20:55:30,087 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 7f864e4ec7cff383a8a9db2371cff040#B#compaction#66 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:30,088 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7f864e4ec7cff383a8a9db2371cff040#A#compaction#67 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:30,089 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:30,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:30,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:30,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:30,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:30,098 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:30,098 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:30,098 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:30,098 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:30,098 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:30,099 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:30,099 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:30,099 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:30,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:30,100 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,100 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:30,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:30,100 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:30,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:30,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:30,100 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:30,102 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:30,102 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:30,102 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:30,102 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:30,102 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:30,110 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:30,110 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:30,110 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:30,110 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:30,110 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7f864e4ec7cff383a8a9db2371cff040 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-08 20:55:30,110 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=A 2023-04-08 20:55:30,111 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:55:30,111 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=B 2023-04-08 20:55:30,111 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:55:30,111 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=C 2023-04-08 20:55:30,111 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:55:30,124 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/a4804a8d6d5045c0b083106ad1eb45b7 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/a4804a8d6d5045c0b083106ad1eb45b7 2023-04-08 20:55:30,126 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=441 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/3a8106c26e7241968c3b41d7745a76a2 2023-04-08 20:55:30,130 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7f864e4ec7cff383a8a9db2371cff040/A of 7f864e4ec7cff383a8a9db2371cff040 into a4804a8d6d5045c0b083106ad1eb45b7(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:30,130 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:30,130 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040., storeName=7f864e4ec7cff383a8a9db2371cff040/A, priority=12, startTime=1680987330075; duration=0sec 2023-04-08 20:55:30,130 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:30,130 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 20:55:30,132 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46914 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 20:55:30,132 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): 7f864e4ec7cff383a8a9db2371cff040/C is initiating minor compaction (all files) 2023-04-08 20:55:30,132 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f864e4ec7cff383a8a9db2371cff040/C in TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:30,132 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/d1ede29d5b8e4957a4d6bf8f8d2f603e, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/ba57cacb1d0c41aa81f3e7a64f2d08ba, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/6d654e05b62a43c4b644ad070b2e2f16, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/892a475f03044232a67b6a0608b0458f] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp, totalSize=45.8 K 2023-04-08 20:55:30,133 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting d1ede29d5b8e4957a4d6bf8f8d2f603e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1680987325565 2023-04-08 20:55:30,133 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting ba57cacb1d0c41aa81f3e7a64f2d08ba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1680987326052 2023-04-08 20:55:30,134 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 6d654e05b62a43c4b644ad070b2e2f16, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1680987326691 2023-04-08 20:55:30,135 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 892a475f03044232a67b6a0608b0458f, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1680987328816 2023-04-08 20:55:30,148 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7f864e4ec7cff383a8a9db2371cff040#C#compaction#70 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:30,502 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/84b0803eb6f1461c90d180fe1979b15f as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/84b0803eb6f1461c90d180fe1979b15f 2023-04-08 20:55:30,507 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7f864e4ec7cff383a8a9db2371cff040/B of 7f864e4ec7cff383a8a9db2371cff040 into 84b0803eb6f1461c90d180fe1979b15f(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:30,507 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:30,507 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040., storeName=7f864e4ec7cff383a8a9db2371cff040/B, priority=12, startTime=1680987330075; duration=0sec 2023-04-08 20:55:30,507 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:30,550 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=441 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/ef1e252792af4ff99981a09d5a86c277 2023-04-08 20:55:30,559 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/f5653d6f0f10400d8a8e6701bc9c52c7 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/f5653d6f0f10400d8a8e6701bc9c52c7 2023-04-08 20:55:30,563 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=441 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/36ace8d0b7ad4de99827caad85222499 2023-04-08 20:55:30,565 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7f864e4ec7cff383a8a9db2371cff040/C of 7f864e4ec7cff383a8a9db2371cff040 into f5653d6f0f10400d8a8e6701bc9c52c7(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:30,565 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:30,565 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040., storeName=7f864e4ec7cff383a8a9db2371cff040/C, priority=12, startTime=1680987330075; duration=0sec 2023-04-08 20:55:30,566 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:30,569 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/3a8106c26e7241968c3b41d7745a76a2 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/3a8106c26e7241968c3b41d7745a76a2 2023-04-08 20:55:30,574 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/3a8106c26e7241968c3b41d7745a76a2, entries=150, sequenceid=441, filesize=11.9 K 2023-04-08 20:55:30,575 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/ef1e252792af4ff99981a09d5a86c277 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/ef1e252792af4ff99981a09d5a86c277 2023-04-08 20:55:30,580 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/ef1e252792af4ff99981a09d5a86c277, entries=150, sequenceid=441, filesize=11.9 K 2023-04-08 20:55:30,581 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/36ace8d0b7ad4de99827caad85222499 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/36ace8d0b7ad4de99827caad85222499 2023-04-08 20:55:30,586 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/36ace8d0b7ad4de99827caad85222499, entries=150, sequenceid=441, filesize=11.9 K 2023-04-08 20:55:30,587 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=0 B/0 for 7f864e4ec7cff383a8a9db2371cff040 in 477ms, sequenceid=441, compaction requested=false 2023-04-08 20:55:30,587 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:30,587 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:30,587 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:30,587 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:30,587 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,589 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:30,589 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:30,590 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,590 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:30,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:30,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:30,590 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:30,590 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:30,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:30,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:30,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:30,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:30,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:30,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b1addff[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:30,592 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:30,592 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:30,593 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:30,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:30,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:30,593 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:30,593 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,593 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:30,593 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:30,593 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:30,594 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:30,594 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:30,594 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:30,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:30,594 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:30,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:30,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:30,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:30,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:30,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:30,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:30,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:30,597 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:30,597 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:30,597 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:30,597 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:30,597 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:30,598 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:30,598 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:30,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:30,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:30,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:30,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:30,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:30,598 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:30,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:30,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:30,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:30,599 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:30,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:30,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:30,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:30,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,600 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:30,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:30,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,602 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,602 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:30,602 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:30,602 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,602 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:30,602 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:30,602 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:30,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:30,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:30,602 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:30,602 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:30,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:30,602 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:30,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:30,602 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:30,602 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:30,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:30,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:30,702 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:30,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:30,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:30,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:30,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:30,705 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:30,705 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:30,705 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:30,705 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:30,706 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:30,706 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:30,707 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:30,707 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:30,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:30,707 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,707 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:30,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:30,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:30,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:30,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:30,708 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:30,708 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:30,708 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:30,708 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:30,708 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:30,708 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:30,708 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:30,708 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:30,708 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:30,708 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:30,708 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:30,708 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:30,708 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:30,708 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:30,709 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,709 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:30,709 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:30,709 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,709 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:30,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:30,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:30,710 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:30,710 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:30,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:30,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:30,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:30,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:30,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:30,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@310da26c[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:30,711 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:30,711 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:30,712 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:30,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:30,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:30,712 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:30,712 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:30,712 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:30,712 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,712 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:30,713 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:30,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:30,713 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:30,713 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:30,713 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:30,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:30,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:30,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:30,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:30,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:30,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:30,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:30,715 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:30,715 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:30,715 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:30,715 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:30,715 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:30,716 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:30,716 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:30,716 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:30,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:30,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:30,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:30,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:30,717 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:30,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:30,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:30,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:30,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,718 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:30,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:30,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,720 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,720 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:30,720 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:30,720 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,720 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:30,720 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:30,720 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:30,720 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:30,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:30,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:30,720 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:30,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:30,720 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:30,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:30,721 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:30,721 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:30,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:30,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:30,821 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:30,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:30,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:30,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:30,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:30,824 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:30,824 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:30,824 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:30,824 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:30,824 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:30,825 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:30,825 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:30,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:30,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:30,825 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:30,826 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,826 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:30,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:30,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:30,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:30,827 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:30,827 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:30,827 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:30,827 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:30,827 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:30,827 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:30,827 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:30,827 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:30,827 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:30,827 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:30,828 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:30,828 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:30,828 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:30,828 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:30,828 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,828 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:30,828 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:30,830 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:30,830 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:30,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:30,830 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:30,830 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:30,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:30,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:30,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:30,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:30,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:30,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bb3208e[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:30,832 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:30,832 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:30,832 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:30,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:30,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:30,832 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:30,832 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,833 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:30,832 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:30,833 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:30,833 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:30,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:30,833 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:30,834 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:30,834 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:30,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:30,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:30,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:30,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:30,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:30,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:30,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:30,838 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:30,838 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:30,838 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:30,838 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:30,838 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:30,839 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:30,839 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:30,839 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:30,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:30,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:30,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:30,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:30,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:30,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:30,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:30,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:30,840 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:30,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:30,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:30,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:30,841 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:30,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:30,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,843 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,843 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:30,843 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:30,843 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:30,843 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:30,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:30,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:30,844 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,844 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:30,844 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:30,844 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:30,844 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:30,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:30,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:30,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:30,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:30,844 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:30,844 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:30,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:30,944 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:30,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:30,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:30,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:30,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:30,948 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:30,948 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:30,948 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:30,948 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:30,949 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:30,949 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:30,949 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,949 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:30,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:30,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:30,950 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,950 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:30,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:30,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:30,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:30,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:30,950 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:30,951 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:30,951 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:30,951 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:30,951 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:30,951 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:30,966 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:30,966 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:30,978 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:30,978 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:30,978 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:30,978 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:30,978 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:30,978 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:30,978 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,978 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:30,978 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:30,980 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,980 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:30,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:30,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:30,980 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:30,980 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:30,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:30,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:30,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:30,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:30,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:30,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ed35dfa[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:30,982 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:30,982 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:30,983 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:30,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:30,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:30,983 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:30,983 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:30,983 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:30,983 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,984 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:30,985 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:30,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:30,986 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:30,986 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:30,986 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:30,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:30,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:30,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:30,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:30,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:30,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:30,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:30,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:30,993 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:30,993 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:30,993 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:30,993 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:30,993 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:30,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:30,995 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7f864e4ec7cff383a8a9db2371cff040 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-08 20:55:30,996 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=A 2023-04-08 20:55:30,996 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:30,996 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=B 2023-04-08 20:55:30,996 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:30,996 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=C 2023-04-08 20:55:30,996 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:31,000 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,000 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,000 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:31,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:31,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:31,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:31,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:31,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:31,002 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:31,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:31,004 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:31,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:31,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:31,007 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:31,007 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,007 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:31,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:31,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:31,007 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:31,007 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,007 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,007 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,007 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:31,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:31,007 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:31,007 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:31,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:31,007 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:31,007 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:31,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:31,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:31,028 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/0ac14865761d4781a722208aa5e61ad6 2023-04-08 20:55:31,056 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/30db78cb10c24afd91734bd031cb3941 2023-04-08 20:55:31,068 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:31,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987391063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:31,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40950 deadline: 1680987391064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:31,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40954 deadline: 1680987391065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,075 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:31,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987391067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,076 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:31,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40978 deadline: 1680987391069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,087 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/dec4f31680d443638305696d103760ce 2023-04-08 20:55:31,095 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/0ac14865761d4781a722208aa5e61ad6 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/0ac14865761d4781a722208aa5e61ad6 2023-04-08 20:55:31,101 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/0ac14865761d4781a722208aa5e61ad6, entries=150, sequenceid=459, filesize=11.9 K 2023-04-08 20:55:31,105 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/30db78cb10c24afd91734bd031cb3941 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/30db78cb10c24afd91734bd031cb3941 2023-04-08 20:55:31,108 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:31,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:31,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:31,115 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/30db78cb10c24afd91734bd031cb3941, entries=150, sequenceid=459, filesize=11.9 K 2023-04-08 20:55:31,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:31,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:31,117 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:31,117 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:31,117 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:31,117 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:31,117 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,117 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:31,118 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:31,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:31,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:31,119 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:31,119 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,119 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:31,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:31,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:31,120 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:31,120 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:31,120 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:31,120 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:31,120 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:31,120 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:31,120 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/dec4f31680d443638305696d103760ce as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/dec4f31680d443638305696d103760ce 2023-04-08 20:55:31,125 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:31,125 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:31,125 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:31,125 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:31,125 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:31,125 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:31,125 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:31,125 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:31,126 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,126 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:31,126 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:31,127 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/dec4f31680d443638305696d103760ce, entries=150, sequenceid=459, filesize=11.9 K 2023-04-08 20:55:31,127 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 7f864e4ec7cff383a8a9db2371cff040 in 132ms, sequenceid=459, compaction requested=true 2023-04-08 20:55:31,128 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:31,128 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:55:31,128 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:31,128 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:31,128 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:55:31,128 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:31,129 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:55:31,129 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): 7f864e4ec7cff383a8a9db2371cff040/A is initiating minor compaction (all files) 2023-04-08 20:55:31,129 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:55:31,129 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f864e4ec7cff383a8a9db2371cff040/A in TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:31,129 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 7f864e4ec7cff383a8a9db2371cff040/B is initiating minor compaction (all files) 2023-04-08 20:55:31,129 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f864e4ec7cff383a8a9db2371cff040/B in TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:31,129 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/a4804a8d6d5045c0b083106ad1eb45b7, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/3a8106c26e7241968c3b41d7745a76a2, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/0ac14865761d4781a722208aa5e61ad6] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp, totalSize=36.4 K 2023-04-08 20:55:31,129 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/84b0803eb6f1461c90d180fe1979b15f, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/ef1e252792af4ff99981a09d5a86c277, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/30db78cb10c24afd91734bd031cb3941] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp, totalSize=36.4 K 2023-04-08 20:55:31,130 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting a4804a8d6d5045c0b083106ad1eb45b7, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1680987326695 2023-04-08 20:55:31,130 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 84b0803eb6f1461c90d180fe1979b15f, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1680987326695 2023-04-08 20:55:31,130 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 3a8106c26e7241968c3b41d7745a76a2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=441, earliestPutTs=1680987328850 2023-04-08 20:55:31,130 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting ef1e252792af4ff99981a09d5a86c277, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=441, earliestPutTs=1680987328850 2023-04-08 20:55:31,131 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 0ac14865761d4781a722208aa5e61ad6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1680987330985 2023-04-08 20:55:31,131 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 30db78cb10c24afd91734bd031cb3941, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1680987330985 2023-04-08 20:55:31,133 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,133 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:31,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:31,133 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,134 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:31,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:31,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:31,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:31,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:31,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:31,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43bb96fa[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:31,136 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:31,136 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,141 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,141 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,141 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:31,141 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:31,141 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:31,141 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:31,142 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:31,142 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:31,142 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:31,143 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:31,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:31,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:31,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:31,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:31,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:31,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:31,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:31,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:31,146 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:31,146 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:31,146 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:31,146 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:31,146 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:31,147 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,147 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,147 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:31,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:31,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:31,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,148 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:31,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:31,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:31,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:31,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:31,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:31,149 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:31,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:31,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,151 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:31,151 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:31,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:31,151 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:31,151 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:31,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:31,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:31,151 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:31,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:31,152 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:31,152 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,152 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,152 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,152 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,152 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:31,152 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:31,153 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 7f864e4ec7cff383a8a9db2371cff040#B#compaction#75 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:31,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:31,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:31,169 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7f864e4ec7cff383a8a9db2371cff040#A#compaction#76 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:31,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680987331061, max timestamp=1680987331171Num uniques 859; 2023-04-08 20:55:31,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 20:55:31,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680987331061, max timestamp=1680987331171Num uniques 859; 2023-04-08 20:55:31,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680987331061, max timestamp=1680987331171Num uniques 859; 2023-04-08 20:55:31,174 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7f864e4ec7cff383a8a9db2371cff040 3/3 column families, dataSize=89.45 KB heapSize=262.11 KB 2023-04-08 20:55:31,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 20:55:31,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 20:55:31,174 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=A 2023-04-08 20:55:31,175 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:55:31,175 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=B 2023-04-08 20:55:31,175 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:55:31,175 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=C 2023-04-08 20:55:31,175 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:55:31,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:31,208 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/19dd6f6031f545d798bd624c81232d5b as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/19dd6f6031f545d798bd624c81232d5b 2023-04-08 20:55:31,216 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7f864e4ec7cff383a8a9db2371cff040/B of 7f864e4ec7cff383a8a9db2371cff040 into 19dd6f6031f545d798bd624c81232d5b(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:31,216 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:31,216 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040., storeName=7f864e4ec7cff383a8a9db2371cff040/B, priority=13, startTime=1680987331128; duration=0sec 2023-04-08 20:55:31,217 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:31,217 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:55:31,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680987331196, max timestamp=1680987331217Num uniques 859; 2023-04-08 20:55:31,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680987331196, max timestamp=1680987331217Num uniques 859; 2023-04-08 20:55:31,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-08 20:55:31,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-08 20:55:31,220 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:55:31,220 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 7f864e4ec7cff383a8a9db2371cff040/C is initiating minor compaction (all files) 2023-04-08 20:55:31,220 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f864e4ec7cff383a8a9db2371cff040/C in TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:31,220 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/f5653d6f0f10400d8a8e6701bc9c52c7, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/36ace8d0b7ad4de99827caad85222499, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/dec4f31680d443638305696d103760ce] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp, totalSize=36.4 K 2023-04-08 20:55:31,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680987331196, max timestamp=1680987331217Num uniques 859; 2023-04-08 20:55:31,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-08 20:55:31,222 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting f5653d6f0f10400d8a8e6701bc9c52c7, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1680987326695 2023-04-08 20:55:31,223 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 36ace8d0b7ad4de99827caad85222499, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=441, earliestPutTs=1680987328850 2023-04-08 20:55:31,223 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting dec4f31680d443638305696d103760ce, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1680987330985 2023-04-08 20:55:31,244 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 7f864e4ec7cff383a8a9db2371cff040#C#compaction#78 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:31,252 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:31,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:31,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:31,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680987331248, max timestamp=1680987331257Num uniques 1168; 2023-04-08 20:55:31,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=7688, suffix heap size=149440,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=168 2023-04-08 20:55:31,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680987331248, max timestamp=1680987331257Num uniques 1168; 2023-04-08 20:55:31,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=6870, suffix heap size=149440,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=150 2023-04-08 20:55:31,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7682, totalHeapSize=17712, min timestamp=1680987331248, max timestamp=1680987331257Num uniques 1168; 2023-04-08 20:55:31,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=53482, new segment data size=7682, suffix heap size=149440,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=168 2023-04-08 20:55:31,271 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/9aae050e2c2c40608eccb16d579b4a3f as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/9aae050e2c2c40608eccb16d579b4a3f 2023-04-08 20:55:31,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:31,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:31,278 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:31,278 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:31,278 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:31,278 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:31,279 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,279 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:31,280 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:31,280 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:31,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:31,280 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=480 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/839c2032a04f441fbaa223dd06d90dc6 2023-04-08 20:55:31,280 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,280 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:31,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:31,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:31,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:31,282 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:31,284 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7f864e4ec7cff383a8a9db2371cff040/A of 7f864e4ec7cff383a8a9db2371cff040 into 9aae050e2c2c40608eccb16d579b4a3f(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:31,284 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:31,284 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040., storeName=7f864e4ec7cff383a8a9db2371cff040/A, priority=13, startTime=1680987331128; duration=0sec 2023-04-08 20:55:31,284 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:31,298 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:31,298 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:31,298 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:31,298 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:31,298 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:31,299 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1680987331274, max timestamp=1680987331296Num uniques 1027; 2023-04-08 20:55:31,299 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=17262, suffix heap size=134776,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=377 2023-04-08 20:55:31,299 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1680987331274, max timestamp=1680987331296Num uniques 1027; 2023-04-08 20:55:31,299 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=17262, suffix heap size=134776,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=377 2023-04-08 20:55:31,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1680987331274, max timestamp=1680987331296Num uniques 1009; 2023-04-08 20:55:31,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=17262, suffix heap size=132904,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=377 2023-04-08 20:55:31,306 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:31,306 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:31,306 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:31,306 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:31,306 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:31,306 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:31,306 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:31,306 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:31,306 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,306 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:31,306 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:31,311 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,311 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:31,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:31,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:31,312 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/50f44ca8c8234473a07f704499c5a085 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/50f44ca8c8234473a07f704499c5a085 2023-04-08 20:55:31,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:31,312 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,312 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:31,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:31,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:31,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:31,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ff94223[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:31,314 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:31,314 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,315 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,315 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:31,315 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:31,315 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:31,315 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,316 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:31,316 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:31,316 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,316 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:31,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:31,316 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:31,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:31,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:31,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:31,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:31,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:31,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:31,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:31,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:31,320 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:31,320 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:31,320 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:31,320 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:31,320 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:31,321 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7f864e4ec7cff383a8a9db2371cff040/C of 7f864e4ec7cff383a8a9db2371cff040 into 50f44ca8c8234473a07f704499c5a085(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:31,321 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:31,321 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040., storeName=7f864e4ec7cff383a8a9db2371cff040/C, priority=13, startTime=1680987331128; duration=0sec 2023-04-08 20:55:31,321 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:31,325 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,325 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,325 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:31,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:31,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:31,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:31,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:31,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:31,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,326 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:31,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:31,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:31,328 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:31,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:31,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,336 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,336 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:31,336 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:31,336 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:31,336 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,336 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,336 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,336 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:31,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:31,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:31,337 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:31,337 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:31,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:31,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:31,337 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:31,337 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:31,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:31,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:31,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=436, cellSize=19966, totalHeapSize=45584, min timestamp=1680987331326, max timestamp=1680987331340Num uniques 1236; 2023-04-08 20:55:31,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=56606, new segment data size=19966, suffix heap size=156512,new segment heap size=45584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1236, new segment cells count=436 2023-04-08 20:55:31,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=436, cellSize=19966, totalHeapSize=45584, min timestamp=1680987331326, max timestamp=1680987331340Num uniques 1236; 2023-04-08 20:55:31,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=56606, new segment data size=19966, suffix heap size=156512,new segment heap size=45584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1236, new segment cells count=436 2023-04-08 20:55:31,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=436, cellSize=19966, totalHeapSize=45584, min timestamp=1680987331326, max timestamp=1680987331340Num uniques 1236; 2023-04-08 20:55:31,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=56606, new segment data size=19966, suffix heap size=156512,new segment heap size=45584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1236, new segment cells count=436 2023-04-08 20:55:31,351 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=480 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/3435b575294c4345bf90513b9894e8cb 2023-04-08 20:55:31,406 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:31,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987391400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,410 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:31,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40954 deadline: 1680987391405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:31,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:31,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40978 deadline: 1680987391405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987391406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,411 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:31,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40950 deadline: 1680987391406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,438 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:31,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:31,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:31,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:31,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:31,440 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:31,440 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:31,441 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:31,441 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:31,441 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,441 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:31,442 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:31,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:31,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:31,442 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:31,442 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,442 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:31,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:31,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:31,443 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:31,443 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:31,443 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:31,443 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:31,443 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:31,443 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:31,443 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:31,443 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:31,443 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:31,443 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:31,443 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:31,443 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:31,443 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:31,443 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:31,443 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,444 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:31,444 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:31,444 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,444 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:31,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:31,445 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:31,445 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:31,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:31,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:31,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:31,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:31,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d3ca602[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:31,447 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:31,447 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,447 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,447 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:31,447 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:31,447 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:31,447 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,447 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:31,448 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,448 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:31,448 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:31,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,448 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:31,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:31,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:31,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:31,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:31,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:31,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:31,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:31,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:31,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:31,451 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:31,451 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:31,451 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:31,451 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:31,452 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:31,453 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,453 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:31,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,453 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:31,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:31,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:31,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:31,453 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:31,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:31,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:31,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:31,455 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:31,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:31,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,457 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,457 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:31,457 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:31,457 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,458 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,458 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,458 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,458 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:31,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:31,458 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:31,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:31,458 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:31,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:31,458 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:31,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:31,458 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:31,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:31,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:31,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:31,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:31,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987391507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,512 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:31,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40954 deadline: 1680987391512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,512 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:31,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40978 deadline: 1680987391512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,514 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:31,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987391513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,515 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:31,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40950 deadline: 1680987391513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,558 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:31,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:31,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:31,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:31,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:31,566 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:31,566 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:31,566 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:31,566 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:31,567 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,567 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:31,568 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:31,568 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:31,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:31,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:31,568 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,568 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:31,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:31,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:31,569 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:31,569 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:31,569 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:31,569 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:31,569 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:31,569 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:31,570 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:31,570 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:31,571 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:31,571 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:31,571 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:31,571 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:31,571 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:31,571 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:31,571 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,571 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:31,571 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:31,572 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,572 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:31,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:31,572 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,572 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:31,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:31,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:31,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:31,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:31,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:31,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29441ad2[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:31,576 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:31,576 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,577 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,577 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:31,577 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:31,577 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:31,577 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,577 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:31,578 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,578 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:31,578 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:31,579 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:31,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:31,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:31,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:31,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:31,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:31,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:31,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:31,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:31,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:31,581 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:31,581 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:31,581 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:31,581 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:31,581 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:31,599 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,599 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:31,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,599 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:31,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:31,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:31,600 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:31,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:31,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:31,602 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:31,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:31,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:31,607 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:31,607 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:31,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:31,607 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:31,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:31,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:31,608 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:31,608 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:31,608 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:31,608 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,608 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:31,608 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:31,608 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:31,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:31,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:31,608 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,608 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,708 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:31,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:31,710 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:31,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:31,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987391709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:31,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:31,715 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:31,715 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:31,716 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:31,716 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:31,716 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:31,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40954 deadline: 1680987391715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:31,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987391716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:31,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40950 deadline: 1680987391717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,721 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,721 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:31,722 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,722 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:31,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:31,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:31,722 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,722 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:31,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:31,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:31,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:31,723 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:31,733 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:31,733 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:31,733 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:31,733 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:31,733 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:31,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:31,734 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:31,734 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:31,734 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:31,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40978 deadline: 1680987391728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,734 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:31,734 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:31,734 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:31,734 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:31,734 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:31,734 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,734 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:31,734 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:31,735 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,735 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:31,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:31,736 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,736 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:31,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:31,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:31,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:31,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:31,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:31,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36d75546[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:31,737 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:31,737 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,738 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,738 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:31,738 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:31,738 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:31,738 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,738 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:31,739 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:31,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:31,739 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:31,739 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:31,739 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:31,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:31,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:31,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:31,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:31,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:31,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:31,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:31,745 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:31,745 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:31,745 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:31,745 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:31,745 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:31,747 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,747 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:31,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,748 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:31,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:31,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:31,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:31,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:31,749 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:31,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:31,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:31,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,750 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:31,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:31,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,752 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:31,752 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:31,752 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:31,753 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:31,753 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:31,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:31,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:31,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:31,753 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:31,753 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,753 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,753 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:31,753 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:31,753 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,753 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:31,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:31,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:31,768 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=480 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/805e8f306b4940e186ee40d6d4844f36 2023-04-08 20:55:31,775 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/839c2032a04f441fbaa223dd06d90dc6 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/839c2032a04f441fbaa223dd06d90dc6 2023-04-08 20:55:31,782 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/839c2032a04f441fbaa223dd06d90dc6, entries=150, sequenceid=480, filesize=11.9 K 2023-04-08 20:55:31,784 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/3435b575294c4345bf90513b9894e8cb as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/3435b575294c4345bf90513b9894e8cb 2023-04-08 20:55:31,790 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/3435b575294c4345bf90513b9894e8cb, entries=150, sequenceid=480, filesize=11.9 K 2023-04-08 20:55:31,791 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/805e8f306b4940e186ee40d6d4844f36 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/805e8f306b4940e186ee40d6d4844f36 2023-04-08 20:55:31,796 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/805e8f306b4940e186ee40d6d4844f36, entries=150, sequenceid=480, filesize=11.9 K 2023-04-08 20:55:31,798 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~66.19 KB/67776, currentSize=167.72 KB/171750 for 7f864e4ec7cff383a8a9db2371cff040 in 624ms, sequenceid=480, compaction requested=false 2023-04-08 20:55:31,798 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:31,853 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:31,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:31,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:31,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:31,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:31,856 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:31,856 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:31,856 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:31,856 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:31,856 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,856 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:31,857 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:31,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:31,857 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:31,857 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:31,857 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:31,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:31,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:31,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:31,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:31,858 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:31,858 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:31,858 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:31,858 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:31,858 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:31,858 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:31,858 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:31,858 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:31,858 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:31,858 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:31,859 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7f864e4ec7cff383a8a9db2371cff040 3/3 column families, dataSize=167.72 KB heapSize=458.63 KB 2023-04-08 20:55:31,859 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=A 2023-04-08 20:55:31,859 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:55:31,859 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=B 2023-04-08 20:55:31,859 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:55:31,859 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=C 2023-04-08 20:55:31,859 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:55:31,873 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=571 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/80e3bc5f3d39409fb1a872d4b8c66a41 2023-04-08 20:55:31,885 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=571 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/bada558d6b814f44954c5e4f8fdafef5 2023-04-08 20:55:32,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:32,017 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:32,022 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:32,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40950 deadline: 1680987392020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:32,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987392023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:32,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40954 deadline: 1680987392023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,025 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:32,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987392025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,038 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:32,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40978 deadline: 1680987392036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,125 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:32,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987392124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,129 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:32,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40954 deadline: 1680987392125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,129 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:32,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987392126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,300 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=571 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/f8cdedf02b1c4dfeb00b3d43e799aefe 2023-04-08 20:55:32,304 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/80e3bc5f3d39409fb1a872d4b8c66a41 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/80e3bc5f3d39409fb1a872d4b8c66a41 2023-04-08 20:55:32,309 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/80e3bc5f3d39409fb1a872d4b8c66a41, entries=150, sequenceid=571, filesize=11.9 K 2023-04-08 20:55:32,310 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/bada558d6b814f44954c5e4f8fdafef5 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/bada558d6b814f44954c5e4f8fdafef5 2023-04-08 20:55:32,315 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/bada558d6b814f44954c5e4f8fdafef5, entries=150, sequenceid=571, filesize=11.9 K 2023-04-08 20:55:32,316 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/f8cdedf02b1c4dfeb00b3d43e799aefe as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/f8cdedf02b1c4dfeb00b3d43e799aefe 2023-04-08 20:55:32,320 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/f8cdedf02b1c4dfeb00b3d43e799aefe, entries=150, sequenceid=571, filesize=11.9 K 2023-04-08 20:55:32,321 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~167.72 KB/171750, heapSize ~458.58 KB/469584, currentSize=26.84 KB/27480 for 7f864e4ec7cff383a8a9db2371cff040 in 463ms, sequenceid=571, compaction requested=true 2023-04-08 20:55:32,321 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:32,321 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:32,321 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:32,322 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:32,322 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,322 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:32,322 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:32,324 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,324 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:32,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:32,326 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:32,326 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:32,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:32,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:32,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:32,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:32,327 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:32,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63eaf1e5[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:32,327 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,328 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,328 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:32,328 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:32,328 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:32,328 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,328 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:32,329 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,329 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:32,329 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:32,329 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:32,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:32,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:32,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:32,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:32,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:32,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:32,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:32,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:32,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:32,332 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:32,332 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:32,332 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:32,332 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:32,332 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:32,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:32,333 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7f864e4ec7cff383a8a9db2371cff040 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-08 20:55:32,333 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,333 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:32,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,333 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:32,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:32,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:32,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:32,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:32,334 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,334 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=A 2023-04-08 20:55:32,334 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:32,334 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=B 2023-04-08 20:55:32,334 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:32,334 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=C 2023-04-08 20:55:32,334 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:32,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:32,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:32,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:32,338 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:32,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:32,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,350 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,350 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:32,350 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:32,350 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:32,350 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:32,350 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,350 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,350 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,350 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:32,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:32,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:32,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:32,350 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:32,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:32,351 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:32,351 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:32,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:32,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:32,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680987332340, max timestamp=1680987332366Num uniques 859; 2023-04-08 20:55:32,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680987332340, max timestamp=1680987332366Num uniques 859; 2023-04-08 20:55:32,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-08 20:55:32,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680987332340, max timestamp=1680987332366Num uniques 859; 2023-04-08 20:55:32,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-08 20:55:32,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-08 20:55:32,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:32,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987392419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:32,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40954 deadline: 1680987392420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,425 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:32,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987392421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,451 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:32,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:32,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:32,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:32,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:32,453 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:32,453 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:32,453 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:32,453 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:32,454 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,454 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:32,454 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:32,454 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:32,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:32,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:32,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:32,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:32,455 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:32,456 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,456 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:32,456 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:32,456 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:32,456 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:32,456 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:32,456 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:32,456 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:32,456 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:32,456 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:32,456 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:32,456 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:32,456 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:32,456 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:32,456 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,456 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:32,456 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:32,457 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,457 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:32,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:32,457 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,457 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:32,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:32,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:32,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:32,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:32,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:32,459 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:32,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b6e472c[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:32,459 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,459 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,459 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:32,460 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:32,460 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:32,460 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,460 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:32,460 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,460 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:32,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:32,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:32,461 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:32,461 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:32,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:32,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:32,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:32,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:32,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:32,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:32,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:32,464 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:32,464 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:32,464 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:32,464 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:32,464 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:32,465 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:32,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:32,465 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,465 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:32,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:32,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:32,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:32,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:32,466 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:32,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:32,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,467 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:32,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:32,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,468 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,468 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:32,468 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:32,468 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:32,468 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:32,468 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,468 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:32,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:32,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:32,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:32,468 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:32,469 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,469 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,469 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:32,469 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:32,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:32,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:32,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:32,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40954 deadline: 1680987392522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:32,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987392522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,527 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:32,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987392526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,527 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:32,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40950 deadline: 1680987392527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,539 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:32,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40978 deadline: 1680987392539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,569 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:32,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:32,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:32,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:32,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:32,572 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:32,572 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:32,572 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:32,572 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:32,572 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,572 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:32,573 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:32,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:32,573 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:32,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:32,573 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,573 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:32,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:32,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:32,574 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:32,576 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:32,576 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:32,576 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:32,576 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:32,576 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:32,576 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:32,576 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:32,576 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:32,576 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:32,576 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:32,576 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:32,576 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:32,576 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:32,576 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,576 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:32,576 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:32,577 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,577 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:32,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:32,578 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,578 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:32,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:32,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:32,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:32,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:32,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:32,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62f54219[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:32,579 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:32,579 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,579 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,579 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:32,579 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:32,579 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,579 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:32,579 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:32,580 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,580 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:32,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:32,580 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:32,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:32,580 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:32,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:32,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:32,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:32,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:32,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:32,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:32,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:32,582 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:32,582 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:32,582 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:32,582 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:32,582 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:32,583 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,583 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:32,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:32,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,583 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:32,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:32,583 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:32,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:32,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:32,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:32,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:32,584 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:32,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:32,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,586 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:32,586 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:32,586 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:32,586 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:32,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:32,586 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:32,586 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:32,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:32,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:32,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:32,586 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,587 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:32,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:32,587 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:32,587 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:32,587 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,587 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,687 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:32,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:32,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:32,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:32,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:32,690 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:32,690 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:32,690 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:32,690 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:32,690 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,690 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:32,691 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,691 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:32,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:32,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:32,691 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,691 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:32,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:32,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:32,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:32,692 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:32,692 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:32,692 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:32,692 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:32,692 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:32,692 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:32,692 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:32,692 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:32,692 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:32,692 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:32,692 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:32,692 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:32,692 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:32,692 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:32,692 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,692 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:32,692 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:32,693 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,693 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:32,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:32,693 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:32,693 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:32,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:32,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:32,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:32,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:32,694 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:32,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e844664[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:32,694 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,695 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,695 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:32,695 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,695 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:32,695 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:32,695 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:32,696 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,696 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:32,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:32,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:32,696 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:32,696 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:32,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:32,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:32,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:32,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:32,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:32,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:32,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:32,697 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:32,698 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:32,698 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:32,698 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:32,698 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:32,699 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,699 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:32,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:32,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,699 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:32,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:32,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:32,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:32,699 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:32,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:32,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:32,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,700 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:32,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:32,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,702 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,702 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:32,702 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:32,702 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:32,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:32,702 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:32,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:32,702 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,702 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,702 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:32,702 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,702 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:32,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:32,702 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:32,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:32,702 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:32,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:32,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:32,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:32,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40954 deadline: 1680987392725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:32,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987392726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,731 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:32,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987392729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,743 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=582 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/a2f1668d5a184aa88dba1d34452793fe 2023-04-08 20:55:32,755 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=582 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/1a963d5e60d14aac83a0d81d83cb9b69 2023-04-08 20:55:32,802 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:32,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:32,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:32,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:32,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:32,805 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:32,805 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:32,805 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:32,805 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:32,805 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,806 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:32,807 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,807 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:32,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:32,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:32,807 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,807 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:32,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:32,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:32,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:32,808 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:32,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:32,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:32,808 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:32,808 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:32,808 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:32,808 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:32,808 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:32,808 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:32,808 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:32,809 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:32,809 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:32,809 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:32,809 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:32,809 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,809 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:32,809 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:32,810 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,810 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:32,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:32,810 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:32,810 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:32,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:32,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:32,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:32,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:32,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ad5e788[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:32,811 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:32,811 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,812 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,812 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:32,812 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:32,812 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:32,812 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,812 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:32,812 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,813 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:32,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:32,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:32,813 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:32,813 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:32,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:32,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:32,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:32,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:32,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:32,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:32,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:32,815 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:32,815 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:32,815 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:32,815 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:32,816 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:32,817 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,817 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:32,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:32,817 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:32,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,817 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:32,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:32,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:32,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:32,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:32,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:32,818 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:32,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:32,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,820 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,820 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:32,820 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:32,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:32,820 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:32,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:32,820 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:32,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:32,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:32,820 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:32,820 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,820 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:32,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:32,820 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,820 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:32,820 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,820 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:32,921 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:32,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:32,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:32,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:32,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:32,923 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:32,923 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:32,923 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:32,923 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:32,924 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,924 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:32,924 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:32,924 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:32,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:32,925 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,925 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:32,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:32,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:32,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:32,925 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:32,925 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:32,925 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:32,926 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:32,926 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:32,926 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:32,926 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:32,926 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:32,926 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:32,926 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:32,926 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:32,926 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:32,926 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:32,926 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:32,926 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,926 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:32,926 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:32,927 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,927 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:32,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:32,927 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,927 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:32,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:32,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:32,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:32,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:32,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:32,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@431b8cd9[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:32,928 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:32,928 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,929 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,929 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:32,929 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:32,929 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:32,929 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,929 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:32,930 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,930 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:32,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:32,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:32,930 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:32,930 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:32,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:32,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:32,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:32,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:32,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:32,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:32,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:32,932 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:32,932 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:32,932 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:32,932 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:32,932 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:32,933 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,933 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:32,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:32,933 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:32,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:32,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:32,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:32,934 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:32,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:32,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:32,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,935 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:32,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:32,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,936 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:32,936 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:32,936 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:32,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:32,936 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:32,936 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:32,936 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:32,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:32,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:32,936 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:32,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:32,936 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:32,936 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:32,937 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:32,937 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:32,937 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:33,033 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:33,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40954 deadline: 1680987393031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,034 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:33,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987393032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,034 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:33,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987393033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,037 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:33,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:33,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:33,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:33,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:33,040 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:33,040 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:33,040 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:33,040 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:33,040 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:33,040 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:33,041 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,041 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:33,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:33,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:33,041 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,041 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:33,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:33,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:33,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:33,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:33,042 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:33,042 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:33,042 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:33,042 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:33,043 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:33,043 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:33,044 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:33,044 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:33,044 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:33,044 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:33,044 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:33,044 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:33,044 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:33,044 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:33,044 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,044 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:33,044 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:33,045 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:33,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:33,045 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:33,045 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:33,045 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:33,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:33,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:33,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:33,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:33,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:33,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bb6eb29[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:33,047 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:33,047 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:33,047 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:33,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:33,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:33,047 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,047 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:33,047 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:33,047 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:33,047 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:33,048 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:33,048 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,048 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:33,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:33,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:33,048 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:33,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:33,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:33,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:33,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:33,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:33,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:33,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:33,050 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:33,050 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:33,050 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:33,050 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:33,050 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:33,051 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:33,051 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:33,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:33,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:33,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:33,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:33,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:33,051 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:33,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:33,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:33,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:33,051 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:33,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:33,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:33,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:33,052 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:33,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:33,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,054 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,054 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:33,054 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:33,054 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:33,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:33,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:33,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:33,054 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:33,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:33,054 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:33,054 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:33,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:33,054 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,054 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:33,055 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:33,055 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:33,055 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:33,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:33,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:33,155 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:33,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:33,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:33,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:33,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:33,158 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:33,158 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:33,158 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:33,158 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:33,158 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:33,158 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:33,159 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:33,159 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:33,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:33,159 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,159 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:33,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:33,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:33,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:33,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:33,160 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:33,160 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:33,160 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:33,160 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:33,160 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:33,160 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:33,161 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:33,161 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:33,161 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:33,161 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:33,161 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:33,161 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:33,161 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:33,161 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:33,161 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,161 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:33,161 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:33,162 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,162 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:33,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:33,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:33,163 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:33,163 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:33,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:33,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:33,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:33,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:33,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:33,165 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:33,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47cd3b5d[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:33,165 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:33,165 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:33,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:33,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:33,165 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:33,165 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:33,165 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,165 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:33,165 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:33,166 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,166 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:33,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,166 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:33,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:33,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:33,166 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:33,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:33,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:33,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:33,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:33,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:33,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:33,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:33,168 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:33,169 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:33,169 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:33,169 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:33,169 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:33,169 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:33,169 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:33,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:33,169 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:33,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:33,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:33,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:33,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:33,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:33,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:33,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:33,170 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:33,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:33,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:33,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:33,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,172 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:33,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:33,172 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=582 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/c01366856fd2499cbedfa6e1685add90 2023-04-08 20:55:33,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,173 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,173 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:33,174 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:33,174 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,174 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:33,174 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:33,174 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:33,174 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:33,174 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:33,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:33,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:33,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:33,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:33,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:33,174 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:33,174 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:33,174 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:33,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:33,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:33,178 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/a2f1668d5a184aa88dba1d34452793fe as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/a2f1668d5a184aa88dba1d34452793fe 2023-04-08 20:55:33,183 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/a2f1668d5a184aa88dba1d34452793fe, entries=300, sequenceid=582, filesize=19.0 K 2023-04-08 20:55:33,184 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/1a963d5e60d14aac83a0d81d83cb9b69 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/1a963d5e60d14aac83a0d81d83cb9b69 2023-04-08 20:55:33,188 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/1a963d5e60d14aac83a0d81d83cb9b69, entries=150, sequenceid=582, filesize=11.9 K 2023-04-08 20:55:33,189 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/c01366856fd2499cbedfa6e1685add90 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/c01366856fd2499cbedfa6e1685add90 2023-04-08 20:55:33,194 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/c01366856fd2499cbedfa6e1685add90, entries=150, sequenceid=582, filesize=11.9 K 2023-04-08 20:55:33,195 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 7f864e4ec7cff383a8a9db2371cff040 in 862ms, sequenceid=582, compaction requested=true 2023-04-08 20:55:33,195 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:33,195 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:33,195 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 20:55:33,195 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 20:55:33,195 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:33,195 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:33,196 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 56916 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 20:55:33,197 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49596 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 20:55:33,197 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): 7f864e4ec7cff383a8a9db2371cff040/A is initiating minor compaction (all files) 2023-04-08 20:55:33,197 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 7f864e4ec7cff383a8a9db2371cff040/B is initiating minor compaction (all files) 2023-04-08 20:55:33,197 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f864e4ec7cff383a8a9db2371cff040/A in TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:33,197 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f864e4ec7cff383a8a9db2371cff040/B in TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:33,197 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/9aae050e2c2c40608eccb16d579b4a3f, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/839c2032a04f441fbaa223dd06d90dc6, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/80e3bc5f3d39409fb1a872d4b8c66a41, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/a2f1668d5a184aa88dba1d34452793fe] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp, totalSize=55.6 K 2023-04-08 20:55:33,197 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/19dd6f6031f545d798bd624c81232d5b, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/3435b575294c4345bf90513b9894e8cb, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/bada558d6b814f44954c5e4f8fdafef5, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/1a963d5e60d14aac83a0d81d83cb9b69] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp, totalSize=48.4 K 2023-04-08 20:55:33,197 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 9aae050e2c2c40608eccb16d579b4a3f, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1680987330985 2023-04-08 20:55:33,197 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 19dd6f6031f545d798bd624c81232d5b, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1680987330985 2023-04-08 20:55:33,197 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 839c2032a04f441fbaa223dd06d90dc6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1680987331061 2023-04-08 20:55:33,198 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 3435b575294c4345bf90513b9894e8cb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1680987331061 2023-04-08 20:55:33,198 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 80e3bc5f3d39409fb1a872d4b8c66a41, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=571, earliestPutTs=1680987331403 2023-04-08 20:55:33,198 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting bada558d6b814f44954c5e4f8fdafef5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=571, earliestPutTs=1680987331403 2023-04-08 20:55:33,198 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting a2f1668d5a184aa88dba1d34452793fe, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=582, earliestPutTs=1680987332020 2023-04-08 20:55:33,199 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 1a963d5e60d14aac83a0d81d83cb9b69, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=582, earliestPutTs=1680987332020 2023-04-08 20:55:33,212 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 7f864e4ec7cff383a8a9db2371cff040#B#compaction#87 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:33,214 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7f864e4ec7cff383a8a9db2371cff040#A#compaction#88 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:33,274 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:33,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:33,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:33,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:33,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:33,277 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:33,277 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:33,277 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:33,277 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:33,277 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:33,277 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:33,278 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,278 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:33,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:33,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:33,278 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,278 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:33,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:33,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:33,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:33,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:33,279 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:33,279 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:33,279 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:33,279 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:33,279 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:33,280 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:33,282 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:33,282 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:33,282 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:33,282 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:33,282 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7f864e4ec7cff383a8a9db2371cff040 3/3 column families, dataSize=134.18 KB heapSize=370.92 KB 2023-04-08 20:55:33,282 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=A 2023-04-08 20:55:33,282 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:55:33,283 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=B 2023-04-08 20:55:33,283 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:55:33,283 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=C 2023-04-08 20:55:33,283 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:55:33,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:33,539 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:33,546 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:33,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40954 deadline: 1680987393542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,548 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:33,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40950 deadline: 1680987393543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,548 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:33,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987393544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,548 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:33,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987393544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,548 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:33,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40978 deadline: 1680987393547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,629 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/7461ed81f9984dacb9792cd66a946cce as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/7461ed81f9984dacb9792cd66a946cce 2023-04-08 20:55:33,630 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/43c9f56cf1e1465baa976e107dca3826 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/43c9f56cf1e1465baa976e107dca3826 2023-04-08 20:55:33,634 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7f864e4ec7cff383a8a9db2371cff040/B of 7f864e4ec7cff383a8a9db2371cff040 into 7461ed81f9984dacb9792cd66a946cce(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:33,634 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:33,634 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040., storeName=7f864e4ec7cff383a8a9db2371cff040/B, priority=12, startTime=1680987333195; duration=0sec 2023-04-08 20:55:33,634 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:33,634 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 20:55:33,635 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7f864e4ec7cff383a8a9db2371cff040/A of 7f864e4ec7cff383a8a9db2371cff040 into 43c9f56cf1e1465baa976e107dca3826(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:33,635 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:33,635 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040., storeName=7f864e4ec7cff383a8a9db2371cff040/A, priority=12, startTime=1680987333195; duration=0sec 2023-04-08 20:55:33,635 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:33,636 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49596 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 20:55:33,636 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 7f864e4ec7cff383a8a9db2371cff040/C is initiating minor compaction (all files) 2023-04-08 20:55:33,636 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f864e4ec7cff383a8a9db2371cff040/C in TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:33,636 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/50f44ca8c8234473a07f704499c5a085, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/805e8f306b4940e186ee40d6d4844f36, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/f8cdedf02b1c4dfeb00b3d43e799aefe, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/c01366856fd2499cbedfa6e1685add90] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp, totalSize=48.4 K 2023-04-08 20:55:33,636 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 50f44ca8c8234473a07f704499c5a085, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1680987330985 2023-04-08 20:55:33,636 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 805e8f306b4940e186ee40d6d4844f36, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1680987331061 2023-04-08 20:55:33,637 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting f8cdedf02b1c4dfeb00b3d43e799aefe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=571, earliestPutTs=1680987331403 2023-04-08 20:55:33,637 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting c01366856fd2499cbedfa6e1685add90, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=582, earliestPutTs=1680987332020 2023-04-08 20:55:33,645 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 7f864e4ec7cff383a8a9db2371cff040#C#compaction#90 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:33,649 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:33,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40954 deadline: 1680987393647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,649 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:33,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40950 deadline: 1680987393649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,650 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:33,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987393650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,650 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:33,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987393650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,692 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=616 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/8bfbe475d10f486686d1f79934f96734 2023-04-08 20:55:33,852 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:33,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40954 deadline: 1680987393851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,852 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:33,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987393851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,852 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:33,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40950 deadline: 1680987393851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:33,855 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:33,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987393852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,059 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/b88d9eee4cec41178f283fab8af56f57 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/b88d9eee4cec41178f283fab8af56f57 2023-04-08 20:55:34,064 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7f864e4ec7cff383a8a9db2371cff040/C of 7f864e4ec7cff383a8a9db2371cff040 into b88d9eee4cec41178f283fab8af56f57(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:34,065 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:34,065 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040., storeName=7f864e4ec7cff383a8a9db2371cff040/C, priority=12, startTime=1680987333195; duration=0sec 2023-04-08 20:55:34,065 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:34,109 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=616 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/de60f994849d4666b9562fdc5a0c438e 2023-04-08 20:55:34,156 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:34,156 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:34,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40954 deadline: 1680987394154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987394154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,156 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:34,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40950 deadline: 1680987394154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,158 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:34,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987394157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,523 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=616 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/a1326d0a7bcf4f0fa2f8969c679f569d 2023-04-08 20:55:34,528 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/8bfbe475d10f486686d1f79934f96734 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/8bfbe475d10f486686d1f79934f96734 2023-04-08 20:55:34,532 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/8bfbe475d10f486686d1f79934f96734, entries=150, sequenceid=616, filesize=11.9 K 2023-04-08 20:55:34,533 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/de60f994849d4666b9562fdc5a0c438e as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/de60f994849d4666b9562fdc5a0c438e 2023-04-08 20:55:34,548 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/de60f994849d4666b9562fdc5a0c438e, entries=150, sequenceid=616, filesize=11.9 K 2023-04-08 20:55:34,549 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/a1326d0a7bcf4f0fa2f8969c679f569d as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/a1326d0a7bcf4f0fa2f8969c679f569d 2023-04-08 20:55:34,553 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/a1326d0a7bcf4f0fa2f8969c679f569d, entries=150, sequenceid=616, filesize=11.9 K 2023-04-08 20:55:34,554 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~370.88 KB/379776, currentSize=53.67 KB/54960 for 7f864e4ec7cff383a8a9db2371cff040 in 1272ms, sequenceid=616, compaction requested=false 2023-04-08 20:55:34,554 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:34,555 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:34,555 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:34,555 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:34,555 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,555 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:34,555 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:34,556 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:34,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:34,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:34,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:34,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:34,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32516d8b[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:34,557 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:34,557 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:34,558 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,558 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:34,562 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:34,562 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:34,562 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:34,562 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:34,562 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:34,562 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:34,562 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:34,566 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:34,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:34,566 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:34,566 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:34,566 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:34,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:34,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:34,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:34,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:34,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:34,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:34,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:34,569 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:34,569 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:34,569 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:34,569 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:34,569 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:34,576 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:34,576 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:34,576 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:34,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:34,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:34,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:34,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:34,577 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:34,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:34,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:34,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:34,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:34,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:34,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:34,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:34,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,579 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:34,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:34,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,583 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,583 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:34,583 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:34,583 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,583 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:34,583 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:34,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:34,583 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:34,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:34,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:34,584 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:34,584 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:34,583 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:34,583 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:34,584 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:34,584 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:34,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:34,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:34,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:34,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:34,660 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7f864e4ec7cff383a8a9db2371cff040 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-08 20:55:34,660 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=A 2023-04-08 20:55:34,660 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:34,661 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=B 2023-04-08 20:55:34,661 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:34,661 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=C 2023-04-08 20:55:34,661 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:34,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:34,684 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:34,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:34,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:34,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:34,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:34,694 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:34,694 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:34,694 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:34,694 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:34,698 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:34,698 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=632 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/eeec7e8b728a4887a5d5deaf81a2513d 2023-04-08 20:55:34,698 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:34,699 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:34,699 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:34,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:34,700 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:34,700 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:34,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:34,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:34,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:34,700 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:34,702 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:34,702 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:34,702 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:34,702 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:34,702 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:34,706 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:34,706 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:34,706 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:34,706 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:34,706 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:34,706 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:34,706 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:34,706 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:34,706 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,710 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:34,710 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:34,711 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,711 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:34,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:34,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:34,711 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:34,711 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:34,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:34,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:34,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:34,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:34,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:34,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2db3425c[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:34,713 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:34,713 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:34,713 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:34,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:34,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:34,714 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:34,714 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:34,714 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:34,714 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,714 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:34,714 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,714 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:34,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:34,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:34,714 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:34,715 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:34,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:34,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:34,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:34,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:34,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:34,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:34,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:34,716 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:34,716 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:34,716 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:34,716 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:34,716 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:34,717 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:34,717 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:34,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:34,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:34,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:34,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:34,717 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:34,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:34,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:34,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:34,717 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:34,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:34,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:34,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:34,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:34,718 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:34,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:34,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,720 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,720 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:34,720 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:34,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:34,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:34,720 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:34,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:34,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:34,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:34,720 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:34,720 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:34,720 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,720 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:34,720 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:34,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:34,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:34,720 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:34,720 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:34,720 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:34,732 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:34,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987394694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1680987334671, max timestamp=1680987334696Num uniques 859; 2023-04-08 20:55:34,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-08 20:55:34,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680987334671, max timestamp=1680987334696Num uniques 859; 2023-04-08 20:55:34,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-08 20:55:34,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680987334671, max timestamp=1680987334696Num uniques 859; 2023-04-08 20:55:34,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-08 20:55:34,756 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:34,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40950 deadline: 1680987394753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,757 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:34,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40954 deadline: 1680987394755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,763 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:34,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987394758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,820 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:34,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:34,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:34,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:34,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:34,823 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:34,823 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:34,823 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:34,823 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:34,823 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:34,823 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:34,824 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,824 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:34,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:34,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:34,824 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,824 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:34,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:34,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:34,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:34,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:34,825 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:34,825 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:34,825 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:34,825 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:34,825 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:34,825 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:34,825 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:34,825 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:34,825 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:34,826 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:34,826 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:34,826 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:34,826 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:34,826 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:34,826 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,826 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:34,826 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:34,832 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:34,832 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:34,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:34,833 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:34,833 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:34,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:34,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:34,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:34,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:34,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:34,834 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:34,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11e00b60[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:34,834 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:34,835 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:34,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:34,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:34,835 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:34,835 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:34,835 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:34,835 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,835 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:34,836 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:34,836 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:34,836 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,836 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:34,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:34,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:34,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:34,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:34,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:34,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:34,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:34,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:34,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:34,838 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:34,838 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:34,838 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:34,838 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:34,838 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:34,839 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:34,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987394836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,839 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:34,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:34,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:34,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:34,839 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:34,839 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:34,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:34,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:34,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:34,840 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:34,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:34,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:34,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:34,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:34,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:34,843 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:34,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:34,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,845 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:34,845 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:34,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:34,845 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,845 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:34,845 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:34,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:34,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:34,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:34,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:34,845 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:34,845 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,845 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:34,845 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:34,845 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:34,845 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:34,845 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:34,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:34,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:34,858 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:34,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40954 deadline: 1680987394858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:34,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40950 deadline: 1680987394859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,866 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:34,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987394865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,945 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:34,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:34,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:34,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:34,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:34,951 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:34,951 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:34,951 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:34,952 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:34,952 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:34,952 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:34,953 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,953 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,953 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:34,953 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:34,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:34,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:34,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:34,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:34,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:34,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:34,954 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:34,954 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:34,954 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:34,954 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:34,954 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:34,954 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:34,954 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:34,955 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:34,955 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:34,955 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:34,955 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:34,955 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:34,955 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:34,955 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:34,955 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,955 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:34,955 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:34,956 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,956 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:34,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:34,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:34,956 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:34,956 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:34,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:34,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:34,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:34,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:34,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:34,957 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:34,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f59ddd2[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:34,957 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:34,958 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:34,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:34,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:34,958 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:34,958 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:34,958 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:34,958 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,958 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:34,959 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,959 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:34,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:34,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:34,959 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:34,959 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:34,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:34,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:34,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:34,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:34,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:34,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:34,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:34,966 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:34,966 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:34,966 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:34,966 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:34,966 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:34,975 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:34,975 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:34,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:34,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:34,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:34,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:34,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:34,975 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:34,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:34,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:34,975 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:34,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:34,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:34,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:34,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:34,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,977 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,983 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:34,983 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:34,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:34,990 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,990 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:34,990 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:34,990 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:34,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:34,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:34,990 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:34,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:34,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:34,990 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:34,990 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:34,990 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:34,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:34,990 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:34,991 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,042 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:35,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987395041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:35,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40954 deadline: 1680987395059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,065 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:35,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40950 deadline: 1680987395063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,068 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:35,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987395068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,091 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:35,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:35,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:35,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:35,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:35,093 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:35,093 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:35,093 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:35,093 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:35,094 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,094 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:35,094 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:35,094 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:35,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:35,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:35,095 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,095 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:35,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:35,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:35,095 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:35,096 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:35,096 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:35,096 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:35,096 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:35,096 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:35,096 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:35,096 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:35,096 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:35,096 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:35,096 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:35,096 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:35,096 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:35,096 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,096 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:35,096 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:35,097 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,097 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:35,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:35,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:35,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:35,097 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:35,097 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:35,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:35,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:35,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:35,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:35,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65dd287[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:35,098 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:35,098 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:35,099 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:35,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:35,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:35,099 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:35,099 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:35,099 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:35,099 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,099 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:35,100 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,100 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:35,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:35,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:35,100 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:35,100 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:35,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:35,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:35,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:35,102 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:35,102 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:35,102 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:35,102 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:35,102 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:35,103 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,103 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:35,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:35,103 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:35,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:35,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:35,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:35,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:35,104 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:35,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:35,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,105 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:35,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:35,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,106 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,107 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:35,107 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:35,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:35,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:35,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:35,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:35,107 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:35,107 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:35,107 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:35,107 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:35,107 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:35,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:35,107 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:35,107 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:35,107 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:35,107 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,127 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=632 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/357c044a66224fadabedaf49f5b2950b 2023-04-08 20:55:35,207 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:35,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:35,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:35,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:35,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:35,210 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:35,210 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:35,210 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:35,210 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:35,210 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,210 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:35,211 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:35,211 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:35,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:35,211 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,211 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:35,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:35,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:35,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:35,212 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:35,212 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:35,212 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:35,212 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:35,212 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:35,212 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:35,212 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:35,212 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:35,212 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:35,212 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:35,212 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:35,213 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:35,213 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:35,213 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:35,213 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,213 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:35,213 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:35,213 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:35,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:35,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:35,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:35,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:35,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:35,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:35,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@314bd1b2[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:35,215 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:35,215 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:35,215 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:35,215 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:35,216 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:35,216 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:35,216 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:35,216 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,216 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:35,216 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:35,216 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:35,216 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,216 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:35,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:35,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:35,217 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:35,217 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:35,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:35,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:35,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:35,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:35,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:35,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:35,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:35,218 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:35,218 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:35,218 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:35,218 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:35,218 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:35,220 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,220 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,220 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:35,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:35,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:35,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:35,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:35,220 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:35,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:35,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:35,228 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:35,234 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,234 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:35,234 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:35,234 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,234 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:35,234 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:35,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:35,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:35,234 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:35,234 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:35,234 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:35,234 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:35,234 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:35,234 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:35,234 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:35,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:35,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:35,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:35,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:35,334 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:35,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:35,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:35,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:35,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:35,337 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:35,337 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:35,337 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:35,337 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:35,338 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,338 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:35,339 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:35,339 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:35,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:35,339 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,339 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:35,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:35,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:35,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:35,340 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:35,340 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:35,340 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:35,340 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:35,340 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:35,340 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:35,340 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:35,340 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:35,340 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:35,340 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:35,340 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:35,340 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:35,340 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:35,340 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:35,340 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,341 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:35,341 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:35,341 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,341 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:35,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:35,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:35,342 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:35,342 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:35,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:35,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:35,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:35,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:35,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:35,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@206177f0[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:35,343 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:35,343 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:35,344 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:35,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:35,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:35,344 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:35,344 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:35,344 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:35,344 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,344 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:35,345 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:35,345 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:35,345 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:35,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:35,345 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:35,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:35,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:35,346 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:35,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:35,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987395345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:35,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:35,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:35,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:35,349 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:35,349 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:35,349 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:35,349 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:35,349 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:35,358 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,358 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:35,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:35,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,358 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:35,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:35,359 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:35,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:35,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:35,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:35,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:35,359 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:35,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:35,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,362 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:35,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40954 deadline: 1680987395362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,366 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,366 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:35,366 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:35,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:35,366 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:35,366 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:35,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:35,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:35,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:35,366 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:35,366 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:35,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:35,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:35,366 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:35,366 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,366 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:35,366 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:35,367 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:35,367 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,367 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:35,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40950 deadline: 1680987395367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,371 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:35,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987395371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,467 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:35,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:35,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:35,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:35,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:35,469 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:35,469 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:35,469 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:35,469 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:35,470 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,470 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:35,470 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:35,470 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:35,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:35,471 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,471 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:35,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:35,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:35,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:35,471 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:35,471 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:35,471 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:35,472 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:35,472 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:35,472 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:35,472 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:35,472 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:35,472 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:35,472 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:35,472 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:35,472 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:35,472 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:35,472 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:35,472 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,472 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:35,472 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:35,473 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,473 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:35,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:35,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:35,473 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:35,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:35,473 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:35,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:35,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:35,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:35,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:35,474 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:35,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51278963[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:35,474 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:35,475 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:35,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:35,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:35,475 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:35,475 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:35,475 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:35,476 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:35,476 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:35,476 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:35,476 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,476 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,476 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:35,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:35,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:35,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:35,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:35,478 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:35,478 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:35,478 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:35,478 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:35,478 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:35,479 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,479 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:35,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:35,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,479 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:35,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:35,479 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:35,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:35,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:35,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:35,480 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:35,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:35,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:35,482 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,482 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:35,482 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:35,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:35,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:35,482 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:35,482 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:35,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:35,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:35,482 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:35,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:35,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:35,482 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:35,482 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,482 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:35,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:35,482 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:35,482 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:35,483 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,539 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=632 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/ee577ea5fd3e4b2bb578dac044893dca 2023-04-08 20:55:35,544 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/eeec7e8b728a4887a5d5deaf81a2513d as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/eeec7e8b728a4887a5d5deaf81a2513d 2023-04-08 20:55:35,548 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/eeec7e8b728a4887a5d5deaf81a2513d, entries=250, sequenceid=632, filesize=16.7 K 2023-04-08 20:55:35,549 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/357c044a66224fadabedaf49f5b2950b as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/357c044a66224fadabedaf49f5b2950b 2023-04-08 20:55:35,555 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/357c044a66224fadabedaf49f5b2950b, entries=150, sequenceid=632, filesize=11.9 K 2023-04-08 20:55:35,556 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/ee577ea5fd3e4b2bb578dac044893dca as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/ee577ea5fd3e4b2bb578dac044893dca 2023-04-08 20:55:35,559 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:35,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40978 deadline: 1680987395557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,561 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4156 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040., hostname=jenkins-hbase20.apache.org,41091,1680987313601, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 20:55:35,561 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/ee577ea5fd3e4b2bb578dac044893dca, entries=150, sequenceid=632, filesize=11.9 K 2023-04-08 20:55:35,562 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=111.03 KB/113692 for 7f864e4ec7cff383a8a9db2371cff040 in 902ms, sequenceid=632, compaction requested=true 2023-04-08 20:55:35,562 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:35,562 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:35,562 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:55:35,562 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:55:35,562 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:35,563 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:35,563 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42426 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:55:35,563 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): 7f864e4ec7cff383a8a9db2371cff040/A is initiating minor compaction (all files) 2023-04-08 20:55:35,564 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:55:35,564 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f864e4ec7cff383a8a9db2371cff040/A in TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:35,564 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 7f864e4ec7cff383a8a9db2371cff040/B is initiating minor compaction (all files) 2023-04-08 20:55:35,564 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/43c9f56cf1e1465baa976e107dca3826, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/8bfbe475d10f486686d1f79934f96734, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/eeec7e8b728a4887a5d5deaf81a2513d] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp, totalSize=41.4 K 2023-04-08 20:55:35,564 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f864e4ec7cff383a8a9db2371cff040/B in TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:35,564 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/7461ed81f9984dacb9792cd66a946cce, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/de60f994849d4666b9562fdc5a0c438e, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/357c044a66224fadabedaf49f5b2950b] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp, totalSize=36.7 K 2023-04-08 20:55:35,564 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 43c9f56cf1e1465baa976e107dca3826, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=582, earliestPutTs=1680987332020 2023-04-08 20:55:35,564 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 7461ed81f9984dacb9792cd66a946cce, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=582, earliestPutTs=1680987332020 2023-04-08 20:55:35,564 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 8bfbe475d10f486686d1f79934f96734, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=616, earliestPutTs=1680987332412 2023-04-08 20:55:35,565 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting de60f994849d4666b9562fdc5a0c438e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=616, earliestPutTs=1680987332412 2023-04-08 20:55:35,565 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting eeec7e8b728a4887a5d5deaf81a2513d, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=632, earliestPutTs=1680987333540 2023-04-08 20:55:35,565 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 357c044a66224fadabedaf49f5b2950b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=632, earliestPutTs=1680987333540 2023-04-08 20:55:35,574 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7f864e4ec7cff383a8a9db2371cff040#A#compaction#96 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:35,575 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 7f864e4ec7cff383a8a9db2371cff040#B#compaction#97 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:35,582 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:35,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:35,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:35,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:35,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:35,591 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:35,591 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:35,591 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:35,591 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:35,591 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,592 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:35,592 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:35,592 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:35,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:35,593 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,593 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:35,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:35,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:35,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:35,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:35,594 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:35,594 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:35,594 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:35,594 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:35,594 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:35,594 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:35,594 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:35,594 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:35,595 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:35,595 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:35,595 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7f864e4ec7cff383a8a9db2371cff040 3/3 column families, dataSize=111.03 KB heapSize=313.63 KB 2023-04-08 20:55:35,596 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=A 2023-04-08 20:55:35,596 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:55:35,596 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=B 2023-04-08 20:55:35,596 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:55:35,596 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=C 2023-04-08 20:55:35,596 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:55:35,598 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/fe2029ac93d941b09196578587d3e34b as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/fe2029ac93d941b09196578587d3e34b 2023-04-08 20:55:35,599 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/3cbe2022a9354ed3900d58c95fc7e7f0 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/3cbe2022a9354ed3900d58c95fc7e7f0 2023-04-08 20:55:35,606 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7f864e4ec7cff383a8a9db2371cff040/B of 7f864e4ec7cff383a8a9db2371cff040 into fe2029ac93d941b09196578587d3e34b(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:35,606 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7f864e4ec7cff383a8a9db2371cff040/A of 7f864e4ec7cff383a8a9db2371cff040 into 3cbe2022a9354ed3900d58c95fc7e7f0(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:35,606 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:35,606 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:35,606 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040., storeName=7f864e4ec7cff383a8a9db2371cff040/B, priority=13, startTime=1680987335562; duration=0sec 2023-04-08 20:55:35,606 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040., storeName=7f864e4ec7cff383a8a9db2371cff040/A, priority=13, startTime=1680987335562; duration=0sec 2023-04-08 20:55:35,606 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:35,606 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:35,606 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:55:35,607 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:55:35,607 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 7f864e4ec7cff383a8a9db2371cff040/C is initiating minor compaction (all files) 2023-04-08 20:55:35,608 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f864e4ec7cff383a8a9db2371cff040/C in TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:35,608 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/b88d9eee4cec41178f283fab8af56f57, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/a1326d0a7bcf4f0fa2f8969c679f569d, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/ee577ea5fd3e4b2bb578dac044893dca] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp, totalSize=36.7 K 2023-04-08 20:55:35,608 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting b88d9eee4cec41178f283fab8af56f57, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=582, earliestPutTs=1680987332020 2023-04-08 20:55:35,608 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting a1326d0a7bcf4f0fa2f8969c679f569d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=616, earliestPutTs=1680987332412 2023-04-08 20:55:35,609 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting ee577ea5fd3e4b2bb578dac044893dca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=632, earliestPutTs=1680987333540 2023-04-08 20:55:35,617 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 7f864e4ec7cff383a8a9db2371cff040#C#compaction#99 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:35,630 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/96d3c31c96ea431f8e64181e7d0a0ed2 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/96d3c31c96ea431f8e64181e7d0a0ed2 2023-04-08 20:55:35,635 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7f864e4ec7cff383a8a9db2371cff040/C of 7f864e4ec7cff383a8a9db2371cff040 into 96d3c31c96ea431f8e64181e7d0a0ed2(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:35,635 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:35,635 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040., storeName=7f864e4ec7cff383a8a9db2371cff040/C, priority=13, startTime=1680987335563; duration=0sec 2023-04-08 20:55:35,635 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:35,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:35,849 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:35,871 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:35,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987395869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,872 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:35,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40950 deadline: 1680987395870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,872 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:35,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40954 deadline: 1680987395871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,874 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:35,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987395873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:35,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987395972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,974 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:35,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40950 deadline: 1680987395973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:35,974 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:35,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40954 deadline: 1680987395973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:36,014 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=665 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/0969c25909a24c509b85cd05203575d5 2023-04-08 20:55:36,176 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:36,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987396175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:36,176 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:36,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40950 deadline: 1680987396175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:36,176 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:36,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40954 deadline: 1680987396175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:36,425 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.39 KB at sequenceid=665 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/e8a358daa648458db858bd32a6a83407 2023-04-08 20:55:36,481 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:36,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40950 deadline: 1680987396478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:36,482 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:36,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987396479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:36,482 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:36,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40954 deadline: 1680987396479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:36,836 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.39 KB at sequenceid=665 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/9cf68e5a6fc04f979705c4f2ae39c828 2023-04-08 20:55:36,841 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/0969c25909a24c509b85cd05203575d5 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/0969c25909a24c509b85cd05203575d5 2023-04-08 20:55:36,847 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/0969c25909a24c509b85cd05203575d5, entries=150, sequenceid=665, filesize=11.9 K 2023-04-08 20:55:36,848 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/e8a358daa648458db858bd32a6a83407 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/e8a358daa648458db858bd32a6a83407 2023-04-08 20:55:36,852 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/e8a358daa648458db858bd32a6a83407, entries=150, sequenceid=665, filesize=11.9 K 2023-04-08 20:55:36,855 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/9cf68e5a6fc04f979705c4f2ae39c828 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/9cf68e5a6fc04f979705c4f2ae39c828 2023-04-08 20:55:36,861 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/9cf68e5a6fc04f979705c4f2ae39c828, entries=150, sequenceid=665, filesize=11.9 K 2023-04-08 20:55:36,862 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~111.03 KB/113692, heapSize ~313.58 KB/321104, currentSize=73.80 KB/75570 for 7f864e4ec7cff383a8a9db2371cff040 in 1267ms, sequenceid=665, compaction requested=false 2023-04-08 20:55:36,862 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:36,862 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:36,862 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:36,862 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:36,863 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:36,863 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:36,863 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:36,864 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:36,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:36,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:36,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:36,864 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:36,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:36,864 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:36,864 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:36,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:36,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:36,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:36,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:36,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:36,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3906b02a[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:36,865 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:36,866 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:36,867 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:36,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:36,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:36,867 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:36,867 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:36,867 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:36,867 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:36,867 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:36,869 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:36,869 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:36,869 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:36,869 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:36,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:36,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:36,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:36,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:36,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:36,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:36,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:36,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:36,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:36,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:36,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:36,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:36,874 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:36,874 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:36,874 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:36,874 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:36,874 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:36,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:36,878 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7f864e4ec7cff383a8a9db2371cff040 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-08 20:55:36,879 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=A 2023-04-08 20:55:36,879 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:36,879 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=B 2023-04-08 20:55:36,879 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:36,879 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=C 2023-04-08 20:55:36,879 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:36,882 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:36,882 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:36,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:36,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:36,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:36,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:36,882 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:36,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:36,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:36,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:36,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:36,883 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:36,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:36,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:36,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:36,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:36,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:36,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:36,892 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:36,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:36,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:36,907 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:36,907 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:36,907 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:36,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:36,907 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:36,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:36,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:36,907 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:36,907 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:36,907 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:36,907 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:36,907 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:36,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:36,907 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:36,907 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:36,907 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:36,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:36,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:36,921 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=683 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/0e91440bff1e48ed9ee98e52bf3b0836 2023-04-08 20:55:36,942 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:36,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987396941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:36,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:36,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40954 deadline: 1680987396984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:36,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:36,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40950 deadline: 1680987396984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:36,989 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:36,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987396986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,008 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:37,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:37,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:37,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:37,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:37,025 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:37,025 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:37,025 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:37,025 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:37,025 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:37,025 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:37,026 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,026 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:37,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:37,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:37,026 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,026 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:37,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:37,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:37,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:37,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:37,028 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:37,041 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:37,041 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:37,041 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:37,041 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:37,041 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:37,042 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:37,043 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:37,043 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:37,043 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:37,043 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:37,043 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:37,043 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:37,043 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:37,043 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,043 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:37,043 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:37,044 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:37,045 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:37,045 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:37,045 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:37,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987397044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,047 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:37,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:37,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:37,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:37,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:37,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:37,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:37,048 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:37,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@562f30f9[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:37,048 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:37,050 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:37,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:37,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:37,050 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,050 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:37,052 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:37,052 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:37,052 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:37,053 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:37,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:37,054 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:37,054 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:37,054 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:37,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:37,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:37,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:37,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:37,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:37,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:37,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:37,056 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:37,057 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:37,057 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:37,057 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:37,057 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:37,058 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:37,058 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:37,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:37,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:37,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:37,058 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:37,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:37,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:37,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:37,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:37,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:37,059 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:37,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:37,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:37,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:37,060 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:37,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:37,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,062 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,062 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:37,062 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:37,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:37,062 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:37,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:37,062 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:37,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:37,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:37,062 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:37,062 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,062 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:37,062 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:37,062 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:37,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:37,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:37,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:37,062 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:37,062 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:37,163 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:37,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:37,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:37,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:37,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:37,168 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:37,168 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:37,168 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:37,168 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:37,169 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:37,169 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:37,170 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:37,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:37,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:37,170 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:37,171 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,171 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:37,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:37,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:37,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:37,171 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:37,174 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:37,174 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:37,174 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:37,174 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:37,174 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:37,186 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:37,186 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:37,186 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:37,186 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:37,186 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:37,186 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:37,186 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:37,186 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:37,186 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,187 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:37,187 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:37,188 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:37,188 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:37,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:37,188 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:37,188 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:37,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:37,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:37,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:37,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:37,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:37,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dda033b[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:37,190 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:37,190 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:37,191 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:37,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:37,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:37,191 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:37,191 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:37,191 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:37,191 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,191 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:37,192 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,192 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:37,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:37,192 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:37,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:37,192 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:37,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:37,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:37,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:37,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:37,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:37,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:37,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:37,195 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:37,195 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:37,195 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:37,195 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:37,195 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:37,195 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:37,196 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:37,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:37,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:37,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:37,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:37,196 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:37,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:37,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:37,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:37,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:37,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:37,196 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:37,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:37,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:37,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,197 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:37,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:37,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,199 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,199 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:37,199 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:37,200 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:37,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:37,200 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:37,200 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:37,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:37,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:37,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:37,200 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,200 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:37,200 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:37,200 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:37,200 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:37,200 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:37,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:37,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:37,247 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:37,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987397246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,300 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:37,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:37,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:37,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:37,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:37,303 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:37,303 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:37,303 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:37,303 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:37,303 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:37,303 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:37,304 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:37,304 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:37,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:37,304 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,304 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:37,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:37,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:37,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:37,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:37,305 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:37,305 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:37,305 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:37,305 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:37,305 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:37,305 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:37,306 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:37,306 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:37,306 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:37,306 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:37,306 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:37,306 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:37,306 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:37,306 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:37,306 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,306 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:37,306 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:37,307 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,307 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:37,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:37,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:37,307 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:37,307 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:37,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:37,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:37,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:37,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:37,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:37,308 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:37,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@105be172[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:37,308 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:37,309 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:37,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:37,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:37,309 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:37,309 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:37,309 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:37,309 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,309 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:37,310 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,310 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:37,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:37,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:37,310 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:37,310 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:37,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:37,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:37,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:37,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:37,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:37,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:37,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:37,311 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:37,311 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:37,312 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:37,312 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:37,312 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:37,313 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:37,313 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:37,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:37,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:37,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:37,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:37,313 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:37,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:37,313 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:37,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:37,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:37,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:37,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:37,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:37,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:37,314 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:37,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:37,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,316 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:37,316 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:37,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:37,316 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:37,316 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:37,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:37,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:37,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:37,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:37,316 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:37,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:37,316 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:37,316 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:37,316 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,316 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:37,316 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:37,317 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:37,317 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:37,342 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=683 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/14f5fa9edaed4079b01f56d49438388d 2023-04-08 20:55:37,352 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=683 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/4b47dd5a919c4e56b9d30cce9b45ac75 2023-04-08 20:55:37,357 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/0e91440bff1e48ed9ee98e52bf3b0836 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/0e91440bff1e48ed9ee98e52bf3b0836 2023-04-08 20:55:37,364 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/0e91440bff1e48ed9ee98e52bf3b0836, entries=200, sequenceid=683, filesize=14.3 K 2023-04-08 20:55:37,364 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/14f5fa9edaed4079b01f56d49438388d as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/14f5fa9edaed4079b01f56d49438388d 2023-04-08 20:55:37,369 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/14f5fa9edaed4079b01f56d49438388d, entries=150, sequenceid=683, filesize=11.9 K 2023-04-08 20:55:37,370 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/4b47dd5a919c4e56b9d30cce9b45ac75 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/4b47dd5a919c4e56b9d30cce9b45ac75 2023-04-08 20:55:37,375 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/4b47dd5a919c4e56b9d30cce9b45ac75, entries=150, sequenceid=683, filesize=11.9 K 2023-04-08 20:55:37,375 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 7f864e4ec7cff383a8a9db2371cff040 in 497ms, sequenceid=683, compaction requested=true 2023-04-08 20:55:37,375 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:37,375 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:37,376 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:55:37,376 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:55:37,376 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:37,376 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:37,376 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40088 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:55:37,377 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:55:37,377 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): 7f864e4ec7cff383a8a9db2371cff040/A is initiating minor compaction (all files) 2023-04-08 20:55:37,377 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 7f864e4ec7cff383a8a9db2371cff040/B is initiating minor compaction (all files) 2023-04-08 20:55:37,377 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f864e4ec7cff383a8a9db2371cff040/A in TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:37,377 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f864e4ec7cff383a8a9db2371cff040/B in TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:37,377 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/3cbe2022a9354ed3900d58c95fc7e7f0, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/0969c25909a24c509b85cd05203575d5, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/0e91440bff1e48ed9ee98e52bf3b0836] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp, totalSize=39.1 K 2023-04-08 20:55:37,377 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/fe2029ac93d941b09196578587d3e34b, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/e8a358daa648458db858bd32a6a83407, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/14f5fa9edaed4079b01f56d49438388d] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp, totalSize=36.8 K 2023-04-08 20:55:37,377 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 3cbe2022a9354ed3900d58c95fc7e7f0, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=632, earliestPutTs=1680987333540 2023-04-08 20:55:37,377 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting fe2029ac93d941b09196578587d3e34b, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=632, earliestPutTs=1680987333540 2023-04-08 20:55:37,377 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 0969c25909a24c509b85cd05203575d5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=665, earliestPutTs=1680987334745 2023-04-08 20:55:37,378 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting e8a358daa648458db858bd32a6a83407, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=665, earliestPutTs=1680987334745 2023-04-08 20:55:37,379 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 0e91440bff1e48ed9ee98e52bf3b0836, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=683, earliestPutTs=1680987335866 2023-04-08 20:55:37,379 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 14f5fa9edaed4079b01f56d49438388d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=683, earliestPutTs=1680987335866 2023-04-08 20:55:37,387 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 7f864e4ec7cff383a8a9db2371cff040#B#compaction#105 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:37,389 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7f864e4ec7cff383a8a9db2371cff040#A#compaction#106 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:37,397 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/931336e9320c43778cf16fed62d7d020 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/931336e9320c43778cf16fed62d7d020 2023-04-08 20:55:37,399 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/163185bf1b0d4ef1add65432662e1b99 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/163185bf1b0d4ef1add65432662e1b99 2023-04-08 20:55:37,404 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7f864e4ec7cff383a8a9db2371cff040/B of 7f864e4ec7cff383a8a9db2371cff040 into 931336e9320c43778cf16fed62d7d020(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:37,404 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:37,404 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7f864e4ec7cff383a8a9db2371cff040/A of 7f864e4ec7cff383a8a9db2371cff040 into 163185bf1b0d4ef1add65432662e1b99(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:37,404 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040., storeName=7f864e4ec7cff383a8a9db2371cff040/B, priority=13, startTime=1680987337376; duration=0sec 2023-04-08 20:55:37,404 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:37,404 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040., storeName=7f864e4ec7cff383a8a9db2371cff040/A, priority=13, startTime=1680987337375; duration=0sec 2023-04-08 20:55:37,404 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:37,404 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:37,404 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:55:37,405 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:55:37,405 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 7f864e4ec7cff383a8a9db2371cff040/C is initiating minor compaction (all files) 2023-04-08 20:55:37,405 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f864e4ec7cff383a8a9db2371cff040/C in TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:37,406 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/96d3c31c96ea431f8e64181e7d0a0ed2, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/9cf68e5a6fc04f979705c4f2ae39c828, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/4b47dd5a919c4e56b9d30cce9b45ac75] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp, totalSize=36.8 K 2023-04-08 20:55:37,406 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 96d3c31c96ea431f8e64181e7d0a0ed2, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=632, earliestPutTs=1680987333540 2023-04-08 20:55:37,406 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 9cf68e5a6fc04f979705c4f2ae39c828, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=665, earliestPutTs=1680987334745 2023-04-08 20:55:37,407 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 4b47dd5a919c4e56b9d30cce9b45ac75, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=683, earliestPutTs=1680987335866 2023-04-08 20:55:37,413 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 7f864e4ec7cff383a8a9db2371cff040#C#compaction#107 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:37,417 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:37,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:37,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:37,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:37,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:37,420 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:37,420 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:37,420 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:37,420 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:37,420 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:37,420 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:37,422 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:37,422 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:37,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:37,422 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,422 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:37,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:37,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:37,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:37,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:37,423 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:37,423 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:37,423 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:37,423 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:37,423 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:37,423 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:37,423 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/757f228d8f7b4625858f9b8516e6d11d as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/757f228d8f7b4625858f9b8516e6d11d 2023-04-08 20:55:37,424 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:37,424 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:37,424 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:37,424 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:37,424 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7f864e4ec7cff383a8a9db2371cff040 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-08 20:55:37,424 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=A 2023-04-08 20:55:37,424 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:37,424 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=B 2023-04-08 20:55:37,424 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:37,424 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=C 2023-04-08 20:55:37,424 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:37,434 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=702 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/85106bbe3bf045ac92b7af850fcb484b 2023-04-08 20:55:37,435 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7f864e4ec7cff383a8a9db2371cff040/C of 7f864e4ec7cff383a8a9db2371cff040 into 757f228d8f7b4625858f9b8516e6d11d(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:37,435 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:37,435 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040., storeName=7f864e4ec7cff383a8a9db2371cff040/C, priority=13, startTime=1680987337376; duration=0sec 2023-04-08 20:55:37,435 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:37,453 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=702 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/3566bc31537b4036934b9c01507eb552 2023-04-08 20:55:37,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:37,553 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:37,588 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:37,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987397584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,690 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:37,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987397689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:37,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987397891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,901 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=702 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/642e80bb911c47c68578ef5d13d89f31 2023-04-08 20:55:37,906 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/85106bbe3bf045ac92b7af850fcb484b as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/85106bbe3bf045ac92b7af850fcb484b 2023-04-08 20:55:37,909 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/85106bbe3bf045ac92b7af850fcb484b, entries=150, sequenceid=702, filesize=11.9 K 2023-04-08 20:55:37,911 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/3566bc31537b4036934b9c01507eb552 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/3566bc31537b4036934b9c01507eb552 2023-04-08 20:55:37,916 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/3566bc31537b4036934b9c01507eb552, entries=150, sequenceid=702, filesize=11.9 K 2023-04-08 20:55:37,917 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/642e80bb911c47c68578ef5d13d89f31 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/642e80bb911c47c68578ef5d13d89f31 2023-04-08 20:55:37,924 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/642e80bb911c47c68578ef5d13d89f31, entries=150, sequenceid=702, filesize=11.9 K 2023-04-08 20:55:37,924 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 7f864e4ec7cff383a8a9db2371cff040 in 500ms, sequenceid=702, compaction requested=false 2023-04-08 20:55:37,924 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:37,924 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:37,924 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:37,925 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:37,925 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,926 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:37,926 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:37,927 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,927 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:37,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:37,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:37,928 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:37,928 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:37,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:37,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:37,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:37,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:37,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:37,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cd084b3[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:37,929 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:37,929 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:37,929 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:37,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:37,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:37,929 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:37,930 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:37,930 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,930 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:37,930 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:37,930 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:37,930 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:37,930 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,930 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:37,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:37,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:37,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:37,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:37,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:37,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:37,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:37,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:37,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:37,932 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:37,932 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:37,932 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:37,932 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:37,932 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:37,933 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:37,933 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:37,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:37,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:37,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:37,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:37,933 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:37,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:37,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:37,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:37,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:37,933 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:37,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:37,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:37,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:37,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,934 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:37,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:37,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,936 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,936 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:37,936 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:37,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:37,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:37,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:37,936 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:37,936 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:37,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:37,936 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:37,936 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:37,936 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:37,936 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:37,936 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:37,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:37,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:37,936 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:37,936 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:37,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:37,988 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7f864e4ec7cff383a8a9db2371cff040 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-08 20:55:37,988 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=A 2023-04-08 20:55:37,988 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:37,988 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=B 2023-04-08 20:55:37,988 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:37,988 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=C 2023-04-08 20:55:37,988 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:38,003 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=719 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/b311ff9634ff4085a026b53de708a0af 2023-04-08 20:55:38,014 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:38,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40950 deadline: 1680987398013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,017 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:38,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40954 deadline: 1680987398014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:38,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987398014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,036 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=719 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/4d16dbd89e1a4154ad8afe67f87794f7 2023-04-08 20:55:38,036 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:38,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:38,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:38,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:38,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:38,041 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:38,041 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:38,041 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:38,041 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:38,042 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:38,042 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:38,043 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:38,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:38,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:38,043 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:38,044 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,044 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:38,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:38,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:38,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:38,045 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:38,047 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:38,047 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:38,047 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:38,047 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:38,047 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:38,047 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:38,048 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:38,048 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:38,048 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:38,048 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:38,048 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:38,049 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:38,049 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:38,049 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,049 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:38,049 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:38,057 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=719 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/524bc14c5b9147a6a11c52022a58793c 2023-04-08 20:55:38,058 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,058 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:38,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:38,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:38,058 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:38,059 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:38,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:38,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:38,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:38,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:38,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:38,062 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:38,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33d7d728[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:38,062 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:38,066 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/b311ff9634ff4085a026b53de708a0af as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/b311ff9634ff4085a026b53de708a0af 2023-04-08 20:55:38,066 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:38,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:38,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:38,067 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:38,067 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:38,067 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:38,067 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,067 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:38,068 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:38,068 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:38,068 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:38,068 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:38,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:38,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:38,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:38,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:38,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:38,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:38,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:38,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:38,072 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:38,072 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:38,072 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:38,072 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:38,072 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:38,073 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:38,073 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:38,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:38,073 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:38,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:38,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:38,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:38,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:38,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:38,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:38,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:38,074 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:38,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:38,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:38,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:38,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:38,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:38,077 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,079 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,079 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:38,079 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:38,079 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:38,079 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,079 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:38,079 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:38,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:38,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:38,079 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:38,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:38,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:38,079 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:38,079 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/b311ff9634ff4085a026b53de708a0af, entries=200, sequenceid=719, filesize=14.3 K 2023-04-08 20:55:38,079 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:38,079 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:38,079 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:38,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:38,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:38,080 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/4d16dbd89e1a4154ad8afe67f87794f7 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/4d16dbd89e1a4154ad8afe67f87794f7 2023-04-08 20:55:38,092 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/4d16dbd89e1a4154ad8afe67f87794f7, entries=150, sequenceid=719, filesize=11.9 K 2023-04-08 20:55:38,093 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/524bc14c5b9147a6a11c52022a58793c as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/524bc14c5b9147a6a11c52022a58793c 2023-04-08 20:55:38,104 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/524bc14c5b9147a6a11c52022a58793c, entries=150, sequenceid=719, filesize=11.9 K 2023-04-08 20:55:38,106 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 7f864e4ec7cff383a8a9db2371cff040 in 118ms, sequenceid=719, compaction requested=true 2023-04-08 20:55:38,106 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:38,106 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:38,107 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:55:38,107 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:55:38,107 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:38,107 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:38,109 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37750 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:55:38,109 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40190 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:55:38,109 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 7f864e4ec7cff383a8a9db2371cff040/B is initiating minor compaction (all files) 2023-04-08 20:55:38,109 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): 7f864e4ec7cff383a8a9db2371cff040/A is initiating minor compaction (all files) 2023-04-08 20:55:38,109 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f864e4ec7cff383a8a9db2371cff040/B in TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:38,109 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f864e4ec7cff383a8a9db2371cff040/A in TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:38,109 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/163185bf1b0d4ef1add65432662e1b99, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/85106bbe3bf045ac92b7af850fcb484b, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/b311ff9634ff4085a026b53de708a0af] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp, totalSize=39.2 K 2023-04-08 20:55:38,109 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/931336e9320c43778cf16fed62d7d020, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/3566bc31537b4036934b9c01507eb552, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/4d16dbd89e1a4154ad8afe67f87794f7] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp, totalSize=36.9 K 2023-04-08 20:55:38,109 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 163185bf1b0d4ef1add65432662e1b99, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=683, earliestPutTs=1680987335866 2023-04-08 20:55:38,109 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 931336e9320c43778cf16fed62d7d020, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=683, earliestPutTs=1680987335866 2023-04-08 20:55:38,110 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 85106bbe3bf045ac92b7af850fcb484b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=702, earliestPutTs=1680987336919 2023-04-08 20:55:38,110 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 3566bc31537b4036934b9c01507eb552, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=702, earliestPutTs=1680987336919 2023-04-08 20:55:38,110 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting b311ff9634ff4085a026b53de708a0af, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=719, earliestPutTs=1680987337576 2023-04-08 20:55:38,110 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 4d16dbd89e1a4154ad8afe67f87794f7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=719, earliestPutTs=1680987337576 2023-04-08 20:55:38,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:38,119 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7f864e4ec7cff383a8a9db2371cff040 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-08 20:55:38,121 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=A 2023-04-08 20:55:38,121 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:38,121 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=B 2023-04-08 20:55:38,121 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:38,121 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=C 2023-04-08 20:55:38,121 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:38,123 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7f864e4ec7cff383a8a9db2371cff040#A#compaction#114 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:38,124 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 7f864e4ec7cff383a8a9db2371cff040#B#compaction#115 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:38,139 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=738 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/16133f5078b74bbab76b3f07ede48fee 2023-04-08 20:55:38,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:38,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987398135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,144 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:38,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40950 deadline: 1680987398140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,145 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:38,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40954 deadline: 1680987398141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,156 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/6533cdcd326e4040aac59ca7371d53f1 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/6533cdcd326e4040aac59ca7371d53f1 2023-04-08 20:55:38,157 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/b9c877a9a59542ea9448785c40a11a5a as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/b9c877a9a59542ea9448785c40a11a5a 2023-04-08 20:55:38,165 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7f864e4ec7cff383a8a9db2371cff040/B of 7f864e4ec7cff383a8a9db2371cff040 into 6533cdcd326e4040aac59ca7371d53f1(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:38,165 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:38,165 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040., storeName=7f864e4ec7cff383a8a9db2371cff040/B, priority=13, startTime=1680987338107; duration=0sec 2023-04-08 20:55:38,165 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:38,165 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:55:38,167 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37750 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:55:38,167 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 7f864e4ec7cff383a8a9db2371cff040/C is initiating minor compaction (all files) 2023-04-08 20:55:38,168 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7f864e4ec7cff383a8a9db2371cff040/C in TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:38,168 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/757f228d8f7b4625858f9b8516e6d11d, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/642e80bb911c47c68578ef5d13d89f31, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/524bc14c5b9147a6a11c52022a58793c] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp, totalSize=36.9 K 2023-04-08 20:55:38,168 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 757f228d8f7b4625858f9b8516e6d11d, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=683, earliestPutTs=1680987335866 2023-04-08 20:55:38,169 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 642e80bb911c47c68578ef5d13d89f31, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=702, earliestPutTs=1680987336919 2023-04-08 20:55:38,170 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 524bc14c5b9147a6a11c52022a58793c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=719, earliestPutTs=1680987337576 2023-04-08 20:55:38,172 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7f864e4ec7cff383a8a9db2371cff040/A of 7f864e4ec7cff383a8a9db2371cff040 into b9c877a9a59542ea9448785c40a11a5a(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:38,173 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:38,173 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040., storeName=7f864e4ec7cff383a8a9db2371cff040/A, priority=13, startTime=1680987338106; duration=0sec 2023-04-08 20:55:38,173 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:38,180 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:38,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:38,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:38,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:38,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:38,190 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:38,190 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:38,190 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:38,190 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:38,191 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:38,191 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:38,192 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,192 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:38,192 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 7f864e4ec7cff383a8a9db2371cff040#C#compaction#118 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:38,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:38,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:38,192 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,192 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:38,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:38,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:38,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:38,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:38,193 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:38,195 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:38,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40934 deadline: 1680987398195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,203 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:38,203 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:38,203 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:38,203 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:38,203 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:38,206 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:38,206 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:38,206 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:38,206 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. started... 2023-04-08 20:55:38,206 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. as already flushing 2023-04-08 20:55:38,206 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:38,206 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:38,206 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:38,206 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,206 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:38,206 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:38,208 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:38,208 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:38,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:38,208 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=738 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/a28b437814fe411882c3525c33c99770 2023-04-08 20:55:38,208 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:38,208 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:38,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:38,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:38,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:38,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:38,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:38,210 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:38,210 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:38,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e39d9b2[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:38,211 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:38,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:38,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:38,211 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:38,211 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:38,211 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:38,213 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:38,213 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:38,213 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:38,213 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,213 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:38,213 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:38,213 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:38,213 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:38,213 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:38,214 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:38,214 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:38,216 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:38,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:38,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:38,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:38,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:38,217 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:38,217 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:38,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:38,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:38,217 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:38,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:38,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:38,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:38,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:38,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:38,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,218 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:38,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:38,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,220 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:38,220 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:38,220 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:38,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:38,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:38,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:38,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:38,220 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:38,221 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:38,221 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,221 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:38,220 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:38,221 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:38,221 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:38,221 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:38,221 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:38,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:38,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:38,223 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1232cedd to 127.0.0.1:52440 2023-04-08 20:55:38,223 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:55:38,224 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7124b8aa to 127.0.0.1:52440 2023-04-08 20:55:38,224 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:55:38,228 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x29fd8399 to 127.0.0.1:52440 2023-04-08 20:55:38,228 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:55:38,228 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7746d7a7 to 127.0.0.1:52440 2023-04-08 20:55:38,228 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:55:38,242 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:38,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987398242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,246 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:38,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40954 deadline: 1680987398246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,247 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:38,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40950 deadline: 1680987398247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,321 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:38,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:38,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:38,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40936 deadline: 1680987398445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,449 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:38,450 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:38,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40950 deadline: 1680987398449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40954 deadline: 1680987398449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:38,639 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=738 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/977c9296506147c0a3b83e99c1ddb61a 2023-04-08 20:55:38,650 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/70c5883e738043768b46d405691a63a4 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/70c5883e738043768b46d405691a63a4 2023-04-08 20:55:38,651 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/16133f5078b74bbab76b3f07ede48fee as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/16133f5078b74bbab76b3f07ede48fee 2023-04-08 20:55:38,658 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/16133f5078b74bbab76b3f07ede48fee, entries=300, sequenceid=738, filesize=19.0 K 2023-04-08 20:55:38,658 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7f864e4ec7cff383a8a9db2371cff040/C of 7f864e4ec7cff383a8a9db2371cff040 into 70c5883e738043768b46d405691a63a4(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:38,658 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:38,658 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040., storeName=7f864e4ec7cff383a8a9db2371cff040/C, priority=13, startTime=1680987338107; duration=0sec 2023-04-08 20:55:38,658 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:38,659 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/a28b437814fe411882c3525c33c99770 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/a28b437814fe411882c3525c33c99770 2023-04-08 20:55:38,664 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/a28b437814fe411882c3525c33c99770, entries=150, sequenceid=738, filesize=11.9 K 2023-04-08 20:55:38,665 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/977c9296506147c0a3b83e99c1ddb61a as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/977c9296506147c0a3b83e99c1ddb61a 2023-04-08 20:55:38,670 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/977c9296506147c0a3b83e99c1ddb61a, entries=150, sequenceid=738, filesize=11.9 K 2023-04-08 20:55:38,670 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 7f864e4ec7cff383a8a9db2371cff040 in 551ms, sequenceid=738, compaction requested=false 2023-04-08 20:55:38,670 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:38,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:38,699 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7f864e4ec7cff383a8a9db2371cff040 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-08 20:55:38,699 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x554a753e to 127.0.0.1:52440 2023-04-08 20:55:38,699 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:55:38,699 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=A 2023-04-08 20:55:38,700 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:38,700 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=B 2023-04-08 20:55:38,700 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:38,700 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=C 2023-04-08 20:55:38,700 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:38,752 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b15648c to 127.0.0.1:52440 2023-04-08 20:55:38,752 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:55:38,756 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ee17908 to 127.0.0.1:52440 2023-04-08 20:55:38,756 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:55:38,757 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0e18c51d to 127.0.0.1:52440 2023-04-08 20:55:38,757 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:55:39,112 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=756 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/7f323c3ae37f404aa0d8371fadfc4780 2023-04-08 20:55:39,531 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=756 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/e72e6bb221ed4e4b864c9361b30181ec 2023-04-08 20:55:39,586 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2398b790 to 127.0.0.1:52440 2023-04-08 20:55:39,586 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:55:39,586 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-08 20:55:39,586 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(394): wrote 126 2023-04-08 20:55:39,586 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(394): wrote 104 2023-04-08 20:55:39,586 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(394): wrote 125 2023-04-08 20:55:39,586 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(394): wrote 159 2023-04-08 20:55:39,586 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(394): wrote 118 2023-04-08 20:55:39,586 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-08 20:55:39,586 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(398): read 5568 2023-04-08 20:55:39,586 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(398): read 5256 2023-04-08 20:55:39,586 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-08 20:55:39,586 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(402): scanned 2353 2023-04-08 20:55:39,586 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(403): verified 7050 rows 2023-04-08 20:55:39,587 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(402): scanned 2334 2023-04-08 20:55:39,587 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(403): verified 7002 rows 2023-04-08 20:55:39,587 INFO [Listener at localhost.localdomain/41823] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-08 20:55:39,587 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1bf45e56 to 127.0.0.1:52440 2023-04-08 20:55:39,587 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:55:39,590 INFO [Listener at localhost.localdomain/41823] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-08 20:55:39,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-08 20:55:39,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-08 20:55:39,606 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680987339606"}]},"ts":"1680987339606"} 2023-04-08 20:55:39,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-08 20:55:39,607 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-08 20:55:39,608 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-08 20:55:39,610 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7f864e4ec7cff383a8a9db2371cff040, UNASSIGN}] 2023-04-08 20:55:39,613 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7f864e4ec7cff383a8a9db2371cff040, UNASSIGN 2023-04-08 20:55:39,613 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=7f864e4ec7cff383a8a9db2371cff040, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:39,614 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987339613"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680987339613"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680987339613"}]},"ts":"1680987339613"} 2023-04-08 20:55:39,615 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601}] 2023-04-08 20:55:39,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-08 20:55:39,774 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:39,774 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 7f864e4ec7cff383a8a9db2371cff040, disabling compactions & flushes 2023-04-08 20:55:39,774 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:39,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-08 20:55:39,952 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=756 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/5db9b633298044f29777aa1acd09c69d 2023-04-08 20:55:39,965 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/7f323c3ae37f404aa0d8371fadfc4780 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/7f323c3ae37f404aa0d8371fadfc4780 2023-04-08 20:55:39,970 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/7f323c3ae37f404aa0d8371fadfc4780, entries=150, sequenceid=756, filesize=11.9 K 2023-04-08 20:55:39,971 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/e72e6bb221ed4e4b864c9361b30181ec as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/e72e6bb221ed4e4b864c9361b30181ec 2023-04-08 20:55:39,976 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/e72e6bb221ed4e4b864c9361b30181ec, entries=150, sequenceid=756, filesize=11.9 K 2023-04-08 20:55:39,977 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/5db9b633298044f29777aa1acd09c69d as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/5db9b633298044f29777aa1acd09c69d 2023-04-08 20:55:39,983 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/5db9b633298044f29777aa1acd09c69d, entries=150, sequenceid=756, filesize=11.9 K 2023-04-08 20:55:39,985 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=26.84 KB/27480 for 7f864e4ec7cff383a8a9db2371cff040 in 1285ms, sequenceid=756, compaction requested=true 2023-04-08 20:55:39,985 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:39,985 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:39,985 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:39,985 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. because compaction request was cancelled 2023-04-08 20:55:39,985 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:39,985 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. after waiting 0 ms 2023-04-08 20:55:39,985 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:39,985 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. because compaction request was cancelled 2023-04-08 20:55:39,985 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 7f864e4ec7cff383a8a9db2371cff040 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-08 20:55:39,985 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:39,985 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:39,985 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=A 2023-04-08 20:55:39,985 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. because compaction request was cancelled 2023-04-08 20:55:39,985 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:39,986 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=B 2023-04-08 20:55:39,986 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:39,986 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7f864e4ec7cff383a8a9db2371cff040, store=C 2023-04-08 20:55:39,986 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:40,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-08 20:55:40,393 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=763 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/ce2b08fc964c4f38b8f85acf0a7f9591 2023-04-08 20:55:40,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-08 20:55:40,813 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=763 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/ae64afefebdf4cb98ae0cc5265553e5f 2023-04-08 20:55:41,231 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=763 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/fb70841584414d3ea22cab766e5e8bcd 2023-04-08 20:55:41,241 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/A/ce2b08fc964c4f38b8f85acf0a7f9591 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/ce2b08fc964c4f38b8f85acf0a7f9591 2023-04-08 20:55:41,245 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/ce2b08fc964c4f38b8f85acf0a7f9591, entries=50, sequenceid=763, filesize=7.1 K 2023-04-08 20:55:41,246 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/B/ae64afefebdf4cb98ae0cc5265553e5f as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/ae64afefebdf4cb98ae0cc5265553e5f 2023-04-08 20:55:41,250 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/ae64afefebdf4cb98ae0cc5265553e5f, entries=50, sequenceid=763, filesize=7.1 K 2023-04-08 20:55:41,251 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/.tmp/C/fb70841584414d3ea22cab766e5e8bcd as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/fb70841584414d3ea22cab766e5e8bcd 2023-04-08 20:55:41,255 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/fb70841584414d3ea22cab766e5e8bcd, entries=50, sequenceid=763, filesize=7.1 K 2023-04-08 20:55:41,255 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 7f864e4ec7cff383a8a9db2371cff040 in 1270ms, sequenceid=763, compaction requested=true 2023-04-08 20:55:41,276 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/dd1d859c7c974090969b8f6c989e9843, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/38397fe5b0fd4bd8ac55392f617c5f29, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/817db19b718a487fa825800605da39f8, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/82314af31af742a48041f3ea33f7cdb8, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/610441322c114074b0bb9bb3a69580fb, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/4e886adf443e4de48ac34650d680ee1b, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/64490481dcbd46d6bc6158e839c52142, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/78779c6aea004a1e88fd4bfe778e3e47, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/5f330c06c12045029afddaab062b12c6, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/97e35320631b4b2f92ecfcec0928f2b9, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/a2e32179002f4e68a3675efef1937bda, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/cf7a7d38606e40c8b11ede7d6b783084, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/dde5c34729654cb9b3159cd0ec7ee8d8, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/42d80e3d72d34d6eb929052fe4a1351e, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/68e6e04f5f314640bf21133ede450022, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/7aecca3687dd4dd19d4fdb9342c59e4c, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/460c334fb52f477090c13d81bc9d58bc, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/6cdb90273ec940e3b758cac50556c382, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/a3c884af255542d892c1b831c04e3780, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/7521ecd8050e4de980d763528924a5f2, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/3a8226dbbec14c518eb59cc8b0a8b377, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/952f047109d24c80803402df88ab8f85, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/a4804a8d6d5045c0b083106ad1eb45b7, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/3a8106c26e7241968c3b41d7745a76a2, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/9aae050e2c2c40608eccb16d579b4a3f, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/0ac14865761d4781a722208aa5e61ad6, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/839c2032a04f441fbaa223dd06d90dc6, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/80e3bc5f3d39409fb1a872d4b8c66a41, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/a2f1668d5a184aa88dba1d34452793fe, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/43c9f56cf1e1465baa976e107dca3826, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/8bfbe475d10f486686d1f79934f96734, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/eeec7e8b728a4887a5d5deaf81a2513d, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/3cbe2022a9354ed3900d58c95fc7e7f0, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/0969c25909a24c509b85cd05203575d5, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/0e91440bff1e48ed9ee98e52bf3b0836, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/163185bf1b0d4ef1add65432662e1b99, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/85106bbe3bf045ac92b7af850fcb484b, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/b311ff9634ff4085a026b53de708a0af] to archive 2023-04-08 20:55:41,277 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 20:55:41,282 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/dd1d859c7c974090969b8f6c989e9843 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/dd1d859c7c974090969b8f6c989e9843 2023-04-08 20:55:41,284 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/38397fe5b0fd4bd8ac55392f617c5f29 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/38397fe5b0fd4bd8ac55392f617c5f29 2023-04-08 20:55:41,286 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/817db19b718a487fa825800605da39f8 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/817db19b718a487fa825800605da39f8 2023-04-08 20:55:41,287 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/82314af31af742a48041f3ea33f7cdb8 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/82314af31af742a48041f3ea33f7cdb8 2023-04-08 20:55:41,289 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/610441322c114074b0bb9bb3a69580fb to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/610441322c114074b0bb9bb3a69580fb 2023-04-08 20:55:41,291 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/4e886adf443e4de48ac34650d680ee1b to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/4e886adf443e4de48ac34650d680ee1b 2023-04-08 20:55:41,292 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/64490481dcbd46d6bc6158e839c52142 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/64490481dcbd46d6bc6158e839c52142 2023-04-08 20:55:41,293 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/78779c6aea004a1e88fd4bfe778e3e47 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/78779c6aea004a1e88fd4bfe778e3e47 2023-04-08 20:55:41,294 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/5f330c06c12045029afddaab062b12c6 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/5f330c06c12045029afddaab062b12c6 2023-04-08 20:55:41,295 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/97e35320631b4b2f92ecfcec0928f2b9 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/97e35320631b4b2f92ecfcec0928f2b9 2023-04-08 20:55:41,296 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/a2e32179002f4e68a3675efef1937bda to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/a2e32179002f4e68a3675efef1937bda 2023-04-08 20:55:41,297 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/cf7a7d38606e40c8b11ede7d6b783084 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/cf7a7d38606e40c8b11ede7d6b783084 2023-04-08 20:55:41,298 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/dde5c34729654cb9b3159cd0ec7ee8d8 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/dde5c34729654cb9b3159cd0ec7ee8d8 2023-04-08 20:55:41,299 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/42d80e3d72d34d6eb929052fe4a1351e to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/42d80e3d72d34d6eb929052fe4a1351e 2023-04-08 20:55:41,300 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/68e6e04f5f314640bf21133ede450022 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/68e6e04f5f314640bf21133ede450022 2023-04-08 20:55:41,301 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/7aecca3687dd4dd19d4fdb9342c59e4c to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/7aecca3687dd4dd19d4fdb9342c59e4c 2023-04-08 20:55:41,302 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/460c334fb52f477090c13d81bc9d58bc to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/460c334fb52f477090c13d81bc9d58bc 2023-04-08 20:55:41,303 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/6cdb90273ec940e3b758cac50556c382 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/6cdb90273ec940e3b758cac50556c382 2023-04-08 20:55:41,304 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/a3c884af255542d892c1b831c04e3780 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/a3c884af255542d892c1b831c04e3780 2023-04-08 20:55:41,305 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/7521ecd8050e4de980d763528924a5f2 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/7521ecd8050e4de980d763528924a5f2 2023-04-08 20:55:41,307 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/3a8226dbbec14c518eb59cc8b0a8b377 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/3a8226dbbec14c518eb59cc8b0a8b377 2023-04-08 20:55:41,308 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/952f047109d24c80803402df88ab8f85 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/952f047109d24c80803402df88ab8f85 2023-04-08 20:55:41,308 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/a4804a8d6d5045c0b083106ad1eb45b7 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/a4804a8d6d5045c0b083106ad1eb45b7 2023-04-08 20:55:41,309 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/3a8106c26e7241968c3b41d7745a76a2 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/3a8106c26e7241968c3b41d7745a76a2 2023-04-08 20:55:41,310 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/9aae050e2c2c40608eccb16d579b4a3f to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/9aae050e2c2c40608eccb16d579b4a3f 2023-04-08 20:55:41,311 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/0ac14865761d4781a722208aa5e61ad6 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/0ac14865761d4781a722208aa5e61ad6 2023-04-08 20:55:41,312 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/839c2032a04f441fbaa223dd06d90dc6 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/839c2032a04f441fbaa223dd06d90dc6 2023-04-08 20:55:41,313 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/80e3bc5f3d39409fb1a872d4b8c66a41 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/80e3bc5f3d39409fb1a872d4b8c66a41 2023-04-08 20:55:41,314 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/a2f1668d5a184aa88dba1d34452793fe to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/a2f1668d5a184aa88dba1d34452793fe 2023-04-08 20:55:41,315 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/43c9f56cf1e1465baa976e107dca3826 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/43c9f56cf1e1465baa976e107dca3826 2023-04-08 20:55:41,316 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/8bfbe475d10f486686d1f79934f96734 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/8bfbe475d10f486686d1f79934f96734 2023-04-08 20:55:41,317 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/eeec7e8b728a4887a5d5deaf81a2513d to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/eeec7e8b728a4887a5d5deaf81a2513d 2023-04-08 20:55:41,318 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/3cbe2022a9354ed3900d58c95fc7e7f0 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/3cbe2022a9354ed3900d58c95fc7e7f0 2023-04-08 20:55:41,320 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/0969c25909a24c509b85cd05203575d5 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/0969c25909a24c509b85cd05203575d5 2023-04-08 20:55:41,321 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/0e91440bff1e48ed9ee98e52bf3b0836 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/0e91440bff1e48ed9ee98e52bf3b0836 2023-04-08 20:55:41,322 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/163185bf1b0d4ef1add65432662e1b99 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/163185bf1b0d4ef1add65432662e1b99 2023-04-08 20:55:41,323 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/85106bbe3bf045ac92b7af850fcb484b to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/85106bbe3bf045ac92b7af850fcb484b 2023-04-08 20:55:41,324 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/b311ff9634ff4085a026b53de708a0af to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/b311ff9634ff4085a026b53de708a0af 2023-04-08 20:55:41,347 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/7a14eaf4f9694083800debf93ee163d9, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/fe704aed132441ce8c5ff2dcf06a2511, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/5ab9cf0d2c294c538b1509859b8e2cdb, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/ef6f9dff16bc4e1b8785f9ceea7c60ed, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/f8dde369eedd4faa9828456fde8dd63e, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/ca911e4d57944ab593bddc555c51fc6c, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/f1e853fdff714a898e3b9ecddd7d4a0e, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/91a869ef29d044419264d73786666285, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/27557d97ffa4483c87ceedc332910c67, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/dcafe900a36549e786086ff43d70182f, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/eb0acd8a6b1548ea81aaa09f53c8bee0, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/4ae78f827ced4a189678ec399d58949b, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/e6a173b6c74046cc8e8e1931bb1b7289, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/0a1467a2fd4a4382b279f613ef17115d, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/120080b2476841b8bec275b690eed642, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/48df75b63a8140b8883823d994ea2172, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/3986a8f8d18345a4bab03eb22e7ac3ee, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/9c3ab4a5ad5d4ef4ac219995a7f34c5d, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/1daca8843c5342f594bd679aa1f47288, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/f92f86b93b474bee9f0a516a3cfd966e, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/dbf60d47ac844db6a8b54ccd413ac49a, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/84b0803eb6f1461c90d180fe1979b15f, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/15f0c8b70cc941dfa19f73e848f801c4, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/ef1e252792af4ff99981a09d5a86c277, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/19dd6f6031f545d798bd624c81232d5b, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/30db78cb10c24afd91734bd031cb3941, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/3435b575294c4345bf90513b9894e8cb, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/bada558d6b814f44954c5e4f8fdafef5, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/7461ed81f9984dacb9792cd66a946cce, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/1a963d5e60d14aac83a0d81d83cb9b69, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/de60f994849d4666b9562fdc5a0c438e, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/fe2029ac93d941b09196578587d3e34b, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/357c044a66224fadabedaf49f5b2950b, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/e8a358daa648458db858bd32a6a83407, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/931336e9320c43778cf16fed62d7d020, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/14f5fa9edaed4079b01f56d49438388d, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/3566bc31537b4036934b9c01507eb552, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/4d16dbd89e1a4154ad8afe67f87794f7] to archive 2023-04-08 20:55:41,348 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 20:55:41,350 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/7a14eaf4f9694083800debf93ee163d9 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/7a14eaf4f9694083800debf93ee163d9 2023-04-08 20:55:41,351 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/fe704aed132441ce8c5ff2dcf06a2511 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/fe704aed132441ce8c5ff2dcf06a2511 2023-04-08 20:55:41,352 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/5ab9cf0d2c294c538b1509859b8e2cdb to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/5ab9cf0d2c294c538b1509859b8e2cdb 2023-04-08 20:55:41,353 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/ef6f9dff16bc4e1b8785f9ceea7c60ed to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/ef6f9dff16bc4e1b8785f9ceea7c60ed 2023-04-08 20:55:41,354 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/f8dde369eedd4faa9828456fde8dd63e to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/f8dde369eedd4faa9828456fde8dd63e 2023-04-08 20:55:41,355 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/ca911e4d57944ab593bddc555c51fc6c to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/ca911e4d57944ab593bddc555c51fc6c 2023-04-08 20:55:41,356 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/f1e853fdff714a898e3b9ecddd7d4a0e to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/f1e853fdff714a898e3b9ecddd7d4a0e 2023-04-08 20:55:41,357 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/91a869ef29d044419264d73786666285 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/91a869ef29d044419264d73786666285 2023-04-08 20:55:41,358 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/27557d97ffa4483c87ceedc332910c67 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/27557d97ffa4483c87ceedc332910c67 2023-04-08 20:55:41,359 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/dcafe900a36549e786086ff43d70182f to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/dcafe900a36549e786086ff43d70182f 2023-04-08 20:55:41,360 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/eb0acd8a6b1548ea81aaa09f53c8bee0 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/eb0acd8a6b1548ea81aaa09f53c8bee0 2023-04-08 20:55:41,361 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/4ae78f827ced4a189678ec399d58949b to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/4ae78f827ced4a189678ec399d58949b 2023-04-08 20:55:41,362 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/e6a173b6c74046cc8e8e1931bb1b7289 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/e6a173b6c74046cc8e8e1931bb1b7289 2023-04-08 20:55:41,363 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/0a1467a2fd4a4382b279f613ef17115d to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/0a1467a2fd4a4382b279f613ef17115d 2023-04-08 20:55:41,364 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/120080b2476841b8bec275b690eed642 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/120080b2476841b8bec275b690eed642 2023-04-08 20:55:41,365 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/48df75b63a8140b8883823d994ea2172 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/48df75b63a8140b8883823d994ea2172 2023-04-08 20:55:41,366 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/3986a8f8d18345a4bab03eb22e7ac3ee to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/3986a8f8d18345a4bab03eb22e7ac3ee 2023-04-08 20:55:41,366 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/9c3ab4a5ad5d4ef4ac219995a7f34c5d to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/9c3ab4a5ad5d4ef4ac219995a7f34c5d 2023-04-08 20:55:41,367 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/1daca8843c5342f594bd679aa1f47288 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/1daca8843c5342f594bd679aa1f47288 2023-04-08 20:55:41,368 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/f92f86b93b474bee9f0a516a3cfd966e to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/f92f86b93b474bee9f0a516a3cfd966e 2023-04-08 20:55:41,369 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/dbf60d47ac844db6a8b54ccd413ac49a to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/dbf60d47ac844db6a8b54ccd413ac49a 2023-04-08 20:55:41,370 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/84b0803eb6f1461c90d180fe1979b15f to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/84b0803eb6f1461c90d180fe1979b15f 2023-04-08 20:55:41,371 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/15f0c8b70cc941dfa19f73e848f801c4 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/15f0c8b70cc941dfa19f73e848f801c4 2023-04-08 20:55:41,372 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/ef1e252792af4ff99981a09d5a86c277 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/ef1e252792af4ff99981a09d5a86c277 2023-04-08 20:55:41,373 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/19dd6f6031f545d798bd624c81232d5b to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/19dd6f6031f545d798bd624c81232d5b 2023-04-08 20:55:41,374 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/30db78cb10c24afd91734bd031cb3941 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/30db78cb10c24afd91734bd031cb3941 2023-04-08 20:55:41,375 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/3435b575294c4345bf90513b9894e8cb to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/3435b575294c4345bf90513b9894e8cb 2023-04-08 20:55:41,376 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/bada558d6b814f44954c5e4f8fdafef5 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/bada558d6b814f44954c5e4f8fdafef5 2023-04-08 20:55:41,376 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/7461ed81f9984dacb9792cd66a946cce to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/7461ed81f9984dacb9792cd66a946cce 2023-04-08 20:55:41,377 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/1a963d5e60d14aac83a0d81d83cb9b69 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/1a963d5e60d14aac83a0d81d83cb9b69 2023-04-08 20:55:41,378 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/de60f994849d4666b9562fdc5a0c438e to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/de60f994849d4666b9562fdc5a0c438e 2023-04-08 20:55:41,379 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/fe2029ac93d941b09196578587d3e34b to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/fe2029ac93d941b09196578587d3e34b 2023-04-08 20:55:41,380 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/357c044a66224fadabedaf49f5b2950b to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/357c044a66224fadabedaf49f5b2950b 2023-04-08 20:55:41,381 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/e8a358daa648458db858bd32a6a83407 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/e8a358daa648458db858bd32a6a83407 2023-04-08 20:55:41,382 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/931336e9320c43778cf16fed62d7d020 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/931336e9320c43778cf16fed62d7d020 2023-04-08 20:55:41,383 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/14f5fa9edaed4079b01f56d49438388d to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/14f5fa9edaed4079b01f56d49438388d 2023-04-08 20:55:41,384 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/3566bc31537b4036934b9c01507eb552 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/3566bc31537b4036934b9c01507eb552 2023-04-08 20:55:41,385 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/4d16dbd89e1a4154ad8afe67f87794f7 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/4d16dbd89e1a4154ad8afe67f87794f7 2023-04-08 20:55:41,390 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/94ca5359441b46eb9e40c675dcb47629, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/bfa7f40724304d84bf7c9210a24ffc08, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/0813ffcd2c724bda8ffb3627e1435e8d, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/c2bb36811fbc407cb1d0e50f718f880f, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/7fa021fe51494aae8eca0d0a5d5de938, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/7d11d98db0f04c30a525c42b67812a2f, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/4cd9ff4a596a445cbc0f3e8f28a93c2c, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/c32c2d2a11814da792fc1eed9ce925fd, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/3229064441c64a68b104f585dbfe3fc7, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/8cebf97792664b31994035e31a78bb75, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/e88d2e0c3f154e189011ab9d2e4c0314, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/6b698b2beb3f4e609d216e733a926c3d, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/855215913b0e4539b88f24f67382b21b, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/dd0d0d2ebd80415689438c87f7fee5e1, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/28fac68d77664a3f9f6bfbede1d7a99f, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/654353f6dded40aabd489a2289fb8553, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/2cfda02a78b44732affba35ca36a3b2e, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/d1ede29d5b8e4957a4d6bf8f8d2f603e, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/863acd97276a4d79aaa602cc8fe42544, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/ba57cacb1d0c41aa81f3e7a64f2d08ba, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/6d654e05b62a43c4b644ad070b2e2f16, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/f5653d6f0f10400d8a8e6701bc9c52c7, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/892a475f03044232a67b6a0608b0458f, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/36ace8d0b7ad4de99827caad85222499, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/50f44ca8c8234473a07f704499c5a085, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/dec4f31680d443638305696d103760ce, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/805e8f306b4940e186ee40d6d4844f36, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/f8cdedf02b1c4dfeb00b3d43e799aefe, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/b88d9eee4cec41178f283fab8af56f57, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/c01366856fd2499cbedfa6e1685add90, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/a1326d0a7bcf4f0fa2f8969c679f569d, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/96d3c31c96ea431f8e64181e7d0a0ed2, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/ee577ea5fd3e4b2bb578dac044893dca, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/9cf68e5a6fc04f979705c4f2ae39c828, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/757f228d8f7b4625858f9b8516e6d11d, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/4b47dd5a919c4e56b9d30cce9b45ac75, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/642e80bb911c47c68578ef5d13d89f31, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/524bc14c5b9147a6a11c52022a58793c] to archive 2023-04-08 20:55:41,391 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 20:55:41,393 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/94ca5359441b46eb9e40c675dcb47629 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/94ca5359441b46eb9e40c675dcb47629 2023-04-08 20:55:41,394 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/bfa7f40724304d84bf7c9210a24ffc08 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/bfa7f40724304d84bf7c9210a24ffc08 2023-04-08 20:55:41,395 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/0813ffcd2c724bda8ffb3627e1435e8d to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/0813ffcd2c724bda8ffb3627e1435e8d 2023-04-08 20:55:41,397 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/c2bb36811fbc407cb1d0e50f718f880f to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/c2bb36811fbc407cb1d0e50f718f880f 2023-04-08 20:55:41,398 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/7fa021fe51494aae8eca0d0a5d5de938 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/7fa021fe51494aae8eca0d0a5d5de938 2023-04-08 20:55:41,399 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/7d11d98db0f04c30a525c42b67812a2f to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/7d11d98db0f04c30a525c42b67812a2f 2023-04-08 20:55:41,401 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/4cd9ff4a596a445cbc0f3e8f28a93c2c to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/4cd9ff4a596a445cbc0f3e8f28a93c2c 2023-04-08 20:55:41,402 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/c32c2d2a11814da792fc1eed9ce925fd to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/c32c2d2a11814da792fc1eed9ce925fd 2023-04-08 20:55:41,403 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/3229064441c64a68b104f585dbfe3fc7 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/3229064441c64a68b104f585dbfe3fc7 2023-04-08 20:55:41,404 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/8cebf97792664b31994035e31a78bb75 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/8cebf97792664b31994035e31a78bb75 2023-04-08 20:55:41,405 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/e88d2e0c3f154e189011ab9d2e4c0314 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/e88d2e0c3f154e189011ab9d2e4c0314 2023-04-08 20:55:41,407 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/6b698b2beb3f4e609d216e733a926c3d to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/6b698b2beb3f4e609d216e733a926c3d 2023-04-08 20:55:41,408 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/855215913b0e4539b88f24f67382b21b to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/855215913b0e4539b88f24f67382b21b 2023-04-08 20:55:41,409 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/dd0d0d2ebd80415689438c87f7fee5e1 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/dd0d0d2ebd80415689438c87f7fee5e1 2023-04-08 20:55:41,410 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/28fac68d77664a3f9f6bfbede1d7a99f to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/28fac68d77664a3f9f6bfbede1d7a99f 2023-04-08 20:55:41,412 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/654353f6dded40aabd489a2289fb8553 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/654353f6dded40aabd489a2289fb8553 2023-04-08 20:55:41,419 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/2cfda02a78b44732affba35ca36a3b2e to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/2cfda02a78b44732affba35ca36a3b2e 2023-04-08 20:55:41,420 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/d1ede29d5b8e4957a4d6bf8f8d2f603e to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/d1ede29d5b8e4957a4d6bf8f8d2f603e 2023-04-08 20:55:41,421 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/863acd97276a4d79aaa602cc8fe42544 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/863acd97276a4d79aaa602cc8fe42544 2023-04-08 20:55:41,422 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/ba57cacb1d0c41aa81f3e7a64f2d08ba to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/ba57cacb1d0c41aa81f3e7a64f2d08ba 2023-04-08 20:55:41,424 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/6d654e05b62a43c4b644ad070b2e2f16 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/6d654e05b62a43c4b644ad070b2e2f16 2023-04-08 20:55:41,424 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/f5653d6f0f10400d8a8e6701bc9c52c7 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/f5653d6f0f10400d8a8e6701bc9c52c7 2023-04-08 20:55:41,425 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/892a475f03044232a67b6a0608b0458f to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/892a475f03044232a67b6a0608b0458f 2023-04-08 20:55:41,435 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/36ace8d0b7ad4de99827caad85222499 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/36ace8d0b7ad4de99827caad85222499 2023-04-08 20:55:41,437 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/50f44ca8c8234473a07f704499c5a085 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/50f44ca8c8234473a07f704499c5a085 2023-04-08 20:55:41,438 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/dec4f31680d443638305696d103760ce to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/dec4f31680d443638305696d103760ce 2023-04-08 20:55:41,440 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/805e8f306b4940e186ee40d6d4844f36 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/805e8f306b4940e186ee40d6d4844f36 2023-04-08 20:55:41,441 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/f8cdedf02b1c4dfeb00b3d43e799aefe to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/f8cdedf02b1c4dfeb00b3d43e799aefe 2023-04-08 20:55:41,443 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/b88d9eee4cec41178f283fab8af56f57 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/b88d9eee4cec41178f283fab8af56f57 2023-04-08 20:55:41,445 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/c01366856fd2499cbedfa6e1685add90 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/c01366856fd2499cbedfa6e1685add90 2023-04-08 20:55:41,446 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/a1326d0a7bcf4f0fa2f8969c679f569d to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/a1326d0a7bcf4f0fa2f8969c679f569d 2023-04-08 20:55:41,448 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/96d3c31c96ea431f8e64181e7d0a0ed2 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/96d3c31c96ea431f8e64181e7d0a0ed2 2023-04-08 20:55:41,449 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/ee577ea5fd3e4b2bb578dac044893dca to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/ee577ea5fd3e4b2bb578dac044893dca 2023-04-08 20:55:41,450 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/9cf68e5a6fc04f979705c4f2ae39c828 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/9cf68e5a6fc04f979705c4f2ae39c828 2023-04-08 20:55:41,451 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/757f228d8f7b4625858f9b8516e6d11d to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/757f228d8f7b4625858f9b8516e6d11d 2023-04-08 20:55:41,452 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/4b47dd5a919c4e56b9d30cce9b45ac75 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/4b47dd5a919c4e56b9d30cce9b45ac75 2023-04-08 20:55:41,453 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/642e80bb911c47c68578ef5d13d89f31 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/642e80bb911c47c68578ef5d13d89f31 2023-04-08 20:55:41,454 DEBUG [StoreCloser-TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/524bc14c5b9147a6a11c52022a58793c to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/524bc14c5b9147a6a11c52022a58793c 2023-04-08 20:55:41,458 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/recovered.edits/766.seqid, newMaxSeqId=766, maxSeqId=1 2023-04-08 20:55:41,459 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040. 2023-04-08 20:55:41,459 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 7f864e4ec7cff383a8a9db2371cff040: 2023-04-08 20:55:41,461 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:41,461 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=7f864e4ec7cff383a8a9db2371cff040, regionState=CLOSED 2023-04-08 20:55:41,462 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987341461"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680987341461"}]},"ts":"1680987341461"} 2023-04-08 20:55:41,465 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-08 20:55:41,465 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 7f864e4ec7cff383a8a9db2371cff040, server=jenkins-hbase20.apache.org,41091,1680987313601 in 1.8480 sec 2023-04-08 20:55:41,466 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-08 20:55:41,466 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=7f864e4ec7cff383a8a9db2371cff040, UNASSIGN in 1.8550 sec 2023-04-08 20:55:41,467 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680987341466"}]},"ts":"1680987341466"} 2023-04-08 20:55:41,468 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-08 20:55:41,468 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-08 20:55:41,471 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.8720 sec 2023-04-08 20:55:41,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-08 20:55:41,715 INFO [Listener at localhost.localdomain/41823] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-08 20:55:41,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-08 20:55:41,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 20:55:41,731 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 20:55:41,732 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 20:55:41,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-08 20:55:41,739 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:41,741 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A, FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B, FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C, FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/recovered.edits] 2023-04-08 20:55:41,744 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/16133f5078b74bbab76b3f07ede48fee to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/16133f5078b74bbab76b3f07ede48fee 2023-04-08 20:55:41,746 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/7f323c3ae37f404aa0d8371fadfc4780 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/7f323c3ae37f404aa0d8371fadfc4780 2023-04-08 20:55:41,747 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/b9c877a9a59542ea9448785c40a11a5a to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/b9c877a9a59542ea9448785c40a11a5a 2023-04-08 20:55:41,748 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/ce2b08fc964c4f38b8f85acf0a7f9591 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/A/ce2b08fc964c4f38b8f85acf0a7f9591 2023-04-08 20:55:41,751 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/6533cdcd326e4040aac59ca7371d53f1 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/6533cdcd326e4040aac59ca7371d53f1 2023-04-08 20:55:41,753 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/a28b437814fe411882c3525c33c99770 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/a28b437814fe411882c3525c33c99770 2023-04-08 20:55:41,754 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/ae64afefebdf4cb98ae0cc5265553e5f to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/ae64afefebdf4cb98ae0cc5265553e5f 2023-04-08 20:55:41,756 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/e72e6bb221ed4e4b864c9361b30181ec to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/B/e72e6bb221ed4e4b864c9361b30181ec 2023-04-08 20:55:41,759 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/5db9b633298044f29777aa1acd09c69d to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/5db9b633298044f29777aa1acd09c69d 2023-04-08 20:55:41,761 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/70c5883e738043768b46d405691a63a4 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/70c5883e738043768b46d405691a63a4 2023-04-08 20:55:41,762 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/977c9296506147c0a3b83e99c1ddb61a to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/977c9296506147c0a3b83e99c1ddb61a 2023-04-08 20:55:41,764 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/fb70841584414d3ea22cab766e5e8bcd to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/C/fb70841584414d3ea22cab766e5e8bcd 2023-04-08 20:55:41,767 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/recovered.edits/766.seqid to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040/recovered.edits/766.seqid 2023-04-08 20:55:41,768 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/7f864e4ec7cff383a8a9db2371cff040 2023-04-08 20:55:41,768 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-08 20:55:41,770 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 20:55:41,788 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-08 20:55:41,826 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-08 20:55:41,827 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 20:55:41,827 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-08 20:55:41,828 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680987341827"}]},"ts":"9223372036854775807"} 2023-04-08 20:55:41,830 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-08 20:55:41,830 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 7f864e4ec7cff383a8a9db2371cff040, NAME => 'TestAcidGuarantees,,1680987316959.7f864e4ec7cff383a8a9db2371cff040.', STARTKEY => '', ENDKEY => ''}] 2023-04-08 20:55:41,830 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-08 20:55:41,830 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680987341830"}]},"ts":"9223372036854775807"} 2023-04-08 20:55:41,832 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-08 20:55:41,833 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 20:55:41,834 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 108 msec 2023-04-08 20:55:41,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-08 20:55:41,836 INFO [Listener at localhost.localdomain/41823] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-08 20:55:41,858 INFO [Listener at localhost.localdomain/41823] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=237 (was 205) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x3244987c-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-124122111_15 at /127.0.0.1:53050 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-124122111_15 at /127.0.0.1:60084 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-124122111_15 at /127.0.0.1:52964 [Waiting for operation #18] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-124122111_15 at /127.0.0.1:52952 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-124122111_15 at /127.0.0.1:53086 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x3244987c-shared-pool-12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x3244987c-shared-pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-124122111_15 at /127.0.0.1:52882 [Waiting for operation #7] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x3244987c-shared-pool-13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase20:41091-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=576 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=235 (was 128) - SystemLoadAverage LEAK? -, ProcessCount=166 (was 175), AvailableMemoryMB=3553 (was 4019) 2023-04-08 20:55:41,871 INFO [Listener at localhost.localdomain/41823] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=237, OpenFileDescriptor=576, MaxFileDescriptor=60000, SystemLoadAverage=235, ProcessCount=166, AvailableMemoryMB=3552 2023-04-08 20:55:41,873 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-08 20:55:41,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-08 20:55:41,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-08 20:55:41,877 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-08 20:55:41,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-08 20:55:41,878 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-08 20:55:41,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-08 20:55:41,887 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:55:41,888 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf empty. 2023-04-08 20:55:41,888 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:55:41,888 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-08 20:55:41,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-08 20:55:42,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-08 20:55:42,305 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-08 20:55:42,309 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 06f3cb8ce0404e826c51a372e2c9bfbf, NAME => 'TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp 2023-04-08 20:55:42,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-08 20:55:42,718 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 20:55:42,719 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 06f3cb8ce0404e826c51a372e2c9bfbf, disabling compactions & flushes 2023-04-08 20:55:42,719 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:42,719 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:42,719 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. after waiting 0 ms 2023-04-08 20:55:42,719 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:42,719 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:42,719 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:55:42,726 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-08 20:55:42,729 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987342729"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680987342729"}]},"ts":"1680987342729"} 2023-04-08 20:55:42,731 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-08 20:55:42,733 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-08 20:55:42,733 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680987342733"}]},"ts":"1680987342733"} 2023-04-08 20:55:42,734 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-08 20:55:42,738 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=06f3cb8ce0404e826c51a372e2c9bfbf, ASSIGN}] 2023-04-08 20:55:42,740 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=06f3cb8ce0404e826c51a372e2c9bfbf, ASSIGN 2023-04-08 20:55:42,741 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=06f3cb8ce0404e826c51a372e2c9bfbf, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,41091,1680987313601; forceNewPlan=false, retain=false 2023-04-08 20:55:42,894 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=06f3cb8ce0404e826c51a372e2c9bfbf, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:42,894 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987342893"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680987342893"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680987342893"}]},"ts":"1680987342893"} 2023-04-08 20:55:42,897 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601}] 2023-04-08 20:55:42,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-08 20:55:43,053 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:43,053 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 06f3cb8ce0404e826c51a372e2c9bfbf, NAME => 'TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.', STARTKEY => '', ENDKEY => ''} 2023-04-08 20:55:43,054 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:55:43,054 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 20:55:43,054 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:55:43,054 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:55:43,056 INFO [StoreOpener-06f3cb8ce0404e826c51a372e2c9bfbf-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:55:43,058 DEBUG [StoreOpener-06f3cb8ce0404e826c51a372e2c9bfbf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A 2023-04-08 20:55:43,058 DEBUG [StoreOpener-06f3cb8ce0404e826c51a372e2c9bfbf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A 2023-04-08 20:55:43,059 INFO [StoreOpener-06f3cb8ce0404e826c51a372e2c9bfbf-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 20:55:43,059 INFO [StoreOpener-06f3cb8ce0404e826c51a372e2c9bfbf-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 06f3cb8ce0404e826c51a372e2c9bfbf columnFamilyName A 2023-04-08 20:55:43,060 INFO [StoreOpener-06f3cb8ce0404e826c51a372e2c9bfbf-1] regionserver.HStore(310): Store=06f3cb8ce0404e826c51a372e2c9bfbf/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 20:55:43,060 INFO [StoreOpener-06f3cb8ce0404e826c51a372e2c9bfbf-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:55:43,062 DEBUG [StoreOpener-06f3cb8ce0404e826c51a372e2c9bfbf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B 2023-04-08 20:55:43,062 DEBUG [StoreOpener-06f3cb8ce0404e826c51a372e2c9bfbf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B 2023-04-08 20:55:43,063 INFO [StoreOpener-06f3cb8ce0404e826c51a372e2c9bfbf-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 20:55:43,063 INFO [StoreOpener-06f3cb8ce0404e826c51a372e2c9bfbf-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 06f3cb8ce0404e826c51a372e2c9bfbf columnFamilyName B 2023-04-08 20:55:43,064 INFO [StoreOpener-06f3cb8ce0404e826c51a372e2c9bfbf-1] regionserver.HStore(310): Store=06f3cb8ce0404e826c51a372e2c9bfbf/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 20:55:43,064 INFO [StoreOpener-06f3cb8ce0404e826c51a372e2c9bfbf-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:55:43,065 DEBUG [StoreOpener-06f3cb8ce0404e826c51a372e2c9bfbf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C 2023-04-08 20:55:43,065 DEBUG [StoreOpener-06f3cb8ce0404e826c51a372e2c9bfbf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C 2023-04-08 20:55:43,066 INFO [StoreOpener-06f3cb8ce0404e826c51a372e2c9bfbf-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 20:55:43,066 INFO [StoreOpener-06f3cb8ce0404e826c51a372e2c9bfbf-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 06f3cb8ce0404e826c51a372e2c9bfbf columnFamilyName C 2023-04-08 20:55:43,066 INFO [StoreOpener-06f3cb8ce0404e826c51a372e2c9bfbf-1] regionserver.HStore(310): Store=06f3cb8ce0404e826c51a372e2c9bfbf/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 20:55:43,066 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:43,067 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:55:43,068 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:55:43,069 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-08 20:55:43,070 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:55:43,071 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-08 20:55:43,072 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 06f3cb8ce0404e826c51a372e2c9bfbf; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70988720, jitterRate=0.05781435966491699}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-08 20:55:43,072 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:55:43,072 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf., pid=18, masterSystemTime=1680987343049 2023-04-08 20:55:43,074 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:43,074 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:43,074 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=06f3cb8ce0404e826c51a372e2c9bfbf, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:43,074 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987343074"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680987343074"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680987343074"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680987343074"}]},"ts":"1680987343074"} 2023-04-08 20:55:43,077 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-08 20:55:43,077 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 in 179 msec 2023-04-08 20:55:43,079 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-08 20:55:43,079 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=06f3cb8ce0404e826c51a372e2c9bfbf, ASSIGN in 339 msec 2023-04-08 20:55:43,080 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-08 20:55:43,080 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680987343080"}]},"ts":"1680987343080"} 2023-04-08 20:55:43,081 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-08 20:55:43,082 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-08 20:55:43,084 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2090 sec 2023-04-08 20:55:43,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-08 20:55:43,985 INFO [Listener at localhost.localdomain/41823] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-08 20:55:43,986 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x10dd04e6 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:55:43,991 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4258ab17, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:55:43,992 DEBUG [Listener at localhost.localdomain/41823] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:55:43,997 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41640, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:55:43,999 DEBUG [Listener at localhost.localdomain/41823] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-08 20:55:44,010 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:46472, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-08 20:55:44,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-08 20:55:44,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-08 20:55:44,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-08 20:55:44,039 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-08 20:55:44,040 DEBUG [PEWorker-5] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-08 20:55:44,040 INFO [PEWorker-5] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-08 20:55:44,045 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-08 20:55:44,057 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=06f3cb8ce0404e826c51a372e2c9bfbf, REOPEN/MOVE}] 2023-04-08 20:55:44,058 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=06f3cb8ce0404e826c51a372e2c9bfbf, REOPEN/MOVE 2023-04-08 20:55:44,058 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=06f3cb8ce0404e826c51a372e2c9bfbf, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,059 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987344058"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680987344058"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680987344058"}]},"ts":"1680987344058"} 2023-04-08 20:55:44,060 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601}] 2023-04-08 20:55:44,212 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:55:44,213 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 06f3cb8ce0404e826c51a372e2c9bfbf, disabling compactions & flushes 2023-04-08 20:55:44,213 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:44,213 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:44,213 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. after waiting 0 ms 2023-04-08 20:55:44,213 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:44,217 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-08 20:55:44,217 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:44,217 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:55:44,217 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 06f3cb8ce0404e826c51a372e2c9bfbf to self. 2023-04-08 20:55:44,219 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:55:44,219 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=06f3cb8ce0404e826c51a372e2c9bfbf, regionState=CLOSED 2023-04-08 20:55:44,219 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987344219"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680987344219"}]},"ts":"1680987344219"} 2023-04-08 20:55:44,222 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-08 20:55:44,222 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 in 161 msec 2023-04-08 20:55:44,223 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=06f3cb8ce0404e826c51a372e2c9bfbf, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,41091,1680987313601; forceNewPlan=false, retain=true 2023-04-08 20:55:44,373 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=06f3cb8ce0404e826c51a372e2c9bfbf, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,373 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987344373"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680987344373"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680987344373"}]},"ts":"1680987344373"} 2023-04-08 20:55:44,375 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601}] 2023-04-08 20:55:44,533 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:44,533 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 06f3cb8ce0404e826c51a372e2c9bfbf, NAME => 'TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.', STARTKEY => '', ENDKEY => ''} 2023-04-08 20:55:44,534 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:55:44,534 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 20:55:44,534 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:55:44,534 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:55:44,539 INFO [StoreOpener-06f3cb8ce0404e826c51a372e2c9bfbf-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:55:44,540 DEBUG [StoreOpener-06f3cb8ce0404e826c51a372e2c9bfbf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A 2023-04-08 20:55:44,540 DEBUG [StoreOpener-06f3cb8ce0404e826c51a372e2c9bfbf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A 2023-04-08 20:55:44,541 INFO [StoreOpener-06f3cb8ce0404e826c51a372e2c9bfbf-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 20:55:44,544 INFO [StoreOpener-06f3cb8ce0404e826c51a372e2c9bfbf-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 06f3cb8ce0404e826c51a372e2c9bfbf columnFamilyName A 2023-04-08 20:55:44,546 INFO [StoreOpener-06f3cb8ce0404e826c51a372e2c9bfbf-1] regionserver.HStore(310): Store=06f3cb8ce0404e826c51a372e2c9bfbf/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 20:55:44,546 INFO [StoreOpener-06f3cb8ce0404e826c51a372e2c9bfbf-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:55:44,548 DEBUG [StoreOpener-06f3cb8ce0404e826c51a372e2c9bfbf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B 2023-04-08 20:55:44,548 DEBUG [StoreOpener-06f3cb8ce0404e826c51a372e2c9bfbf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B 2023-04-08 20:55:44,548 INFO [StoreOpener-06f3cb8ce0404e826c51a372e2c9bfbf-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 20:55:44,549 INFO [StoreOpener-06f3cb8ce0404e826c51a372e2c9bfbf-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 06f3cb8ce0404e826c51a372e2c9bfbf columnFamilyName B 2023-04-08 20:55:44,550 INFO [StoreOpener-06f3cb8ce0404e826c51a372e2c9bfbf-1] regionserver.HStore(310): Store=06f3cb8ce0404e826c51a372e2c9bfbf/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 20:55:44,550 INFO [StoreOpener-06f3cb8ce0404e826c51a372e2c9bfbf-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:55:44,551 DEBUG [StoreOpener-06f3cb8ce0404e826c51a372e2c9bfbf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C 2023-04-08 20:55:44,551 DEBUG [StoreOpener-06f3cb8ce0404e826c51a372e2c9bfbf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C 2023-04-08 20:55:44,551 INFO [StoreOpener-06f3cb8ce0404e826c51a372e2c9bfbf-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 20:55:44,552 INFO [StoreOpener-06f3cb8ce0404e826c51a372e2c9bfbf-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 06f3cb8ce0404e826c51a372e2c9bfbf columnFamilyName C 2023-04-08 20:55:44,553 INFO [StoreOpener-06f3cb8ce0404e826c51a372e2c9bfbf-1] regionserver.HStore(310): Store=06f3cb8ce0404e826c51a372e2c9bfbf/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 20:55:44,553 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:44,554 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:55:44,555 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:55:44,558 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-08 20:55:44,560 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:55:44,561 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 06f3cb8ce0404e826c51a372e2c9bfbf; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66573570, jitterRate=-0.007976502180099487}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-08 20:55:44,561 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:55:44,562 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf., pid=23, masterSystemTime=1680987344527 2023-04-08 20:55:44,563 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:44,563 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:44,564 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=06f3cb8ce0404e826c51a372e2c9bfbf, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,564 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987344564"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680987344564"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680987344564"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680987344564"}]},"ts":"1680987344564"} 2023-04-08 20:55:44,568 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-08 20:55:44,568 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 in 191 msec 2023-04-08 20:55:44,570 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-08 20:55:44,570 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=06f3cb8ce0404e826c51a372e2c9bfbf, REOPEN/MOVE in 511 msec 2023-04-08 20:55:44,573 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-08 20:55:44,573 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 528 msec 2023-04-08 20:55:44,576 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 552 msec 2023-04-08 20:55:44,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-08 20:55:44,586 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x6dc16927 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:55:44,592 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@65dbf2b7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:55:44,592 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x029d68f7 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:55:44,601 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@79385b7c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:55:44,601 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x6322d500 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:55:44,609 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a11c343, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:55:44,610 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x4af56759 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:55:44,617 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4e7aae9f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:55:44,617 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x5d07b2f5 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:55:44,625 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3ca9d9a3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:55:44,626 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x34b669b1 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:55:44,633 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@46a0ca45, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:55:44,634 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x6683e622 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:55:44,638 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@61fff0a2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:55:44,638 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x615d8f34 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:55:44,650 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1896d7cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:55:44,650 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x4890ce53 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:55:44,658 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@45b9cc5a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:55:44,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:44,670 DEBUG [hconnection-0x4aee4fc0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:55:44,670 DEBUG [hconnection-0x4223c11c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:55:44,671 DEBUG [hconnection-0x6a18ce48-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:55:44,671 DEBUG [hconnection-0x5219781b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:55:44,672 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41644, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:55:44,673 DEBUG [hconnection-0x1736b51a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:55:44,674 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41688, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:55:44,674 DEBUG [hconnection-0x28f76201-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:55:44,682 DEBUG [hconnection-0x7cede1ff-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:55:44,682 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41676, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:55:44,682 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41660, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:55:44,683 DEBUG [hconnection-0x6452a79e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:55:44,683 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41694, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:55:44,684 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41716, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:55:44,684 DEBUG [hconnection-0x13010b9b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:55:44,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:44,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:44,691 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:44,693 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:44,693 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:44,693 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:44,693 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41702, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:55:44,693 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41730, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:55:44,694 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:44,696 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:44,700 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:44,702 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41740, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:55:44,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:44,704 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:44,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:44,707 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,707 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:44,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:44,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:44,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:44,711 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:44,712 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:44,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:44,713 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 06f3cb8ce0404e826c51a372e2c9bfbf 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-08 20:55:44,713 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:44,713 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:44,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:55:44,713 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=A 2023-04-08 20:55:44,714 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:44,714 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=B 2023-04-08 20:55:44,714 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:44,714 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=C 2023-04-08 20:55:44,714 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:44,713 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:44,718 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:44,718 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:44,719 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:44,719 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:55:44,719 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:55:44,719 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:44,719 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:44,719 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:44,719 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,734 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:44,734 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:44,737 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,738 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:44,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:44,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:44,738 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:44,738 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:44,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:44,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:44,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:44,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:44,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:44,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51a6eb76[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:44,741 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:44,741 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:44,745 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680987344726, max timestamp=1680987344741Num uniques 859; 2023-04-08 20:55:44,746 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-08 20:55:44,745 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680987344726, max timestamp=1680987344741Num uniques 859; 2023-04-08 20:55:44,745 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680987344726, max timestamp=1680987344741Num uniques 859; 2023-04-08 20:55:44,746 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-08 20:55:44,746 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-08 20:55:44,749 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:44,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:44,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:44,750 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:44,750 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:44,750 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:44,751 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,751 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:44,757 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,757 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:44,757 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:44,757 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:44,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:44,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:44,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:44,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:44,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:44,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:44,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:44,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:44,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:44,760 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:44,760 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:44,760 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:44,760 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:44,760 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:44,766 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:44,766 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:44,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:44,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:44,766 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:44,767 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:44,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:44,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:44,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:44,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:44,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:44,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:44,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:44,769 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,769 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:44,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41716 deadline: 1680987404759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:44,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:44,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:44,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41702 deadline: 1680987404765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:44,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41688 deadline: 1680987404767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:44,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41660 deadline: 1680987404768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,774 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:44,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41694 deadline: 1680987404770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,775 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:44,775 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:44,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:44,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:44,775 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:44,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:44,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:44,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:44,775 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,775 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:44,775 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:44,775 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,775 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:44,775 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:44,775 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:44,775 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:44,775 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:44,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:44,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:44,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:44,822 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304086cb0203405294211b8725611bc445543 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304086cb0203405294211b8725611bc445543 2023-04-08 20:55:44,866 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=14, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/3c80dc546b57474aa05130339c354eda 2023-04-08 20:55:44,876 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:44,876 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:44,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41716 deadline: 1680987404873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,876 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:44,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41660 deadline: 1680987404875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:44,877 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:44,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41688 deadline: 1680987404875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:44,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41702 deadline: 1680987404875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,879 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:44,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41694 deadline: 1680987404876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:44,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:44,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:44,922 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:44,922 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:44,922 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:44,922 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:44,923 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:44,923 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:44,923 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:44,923 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:44,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:44,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:44,924 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,924 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:44,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:44,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:44,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:44,924 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:44,933 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:44,933 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:44,933 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:44,933 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:44,933 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:44,933 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/5f043b692e3746d5a41bfd2c35c6d92d 2023-04-08 20:55:44,933 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:44,933 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:44,933 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:44,933 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:55:44,934 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:55:44,934 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:44,934 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:44,934 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:44,934 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,934 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:44,934 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:44,941 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:44,941 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:44,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:44,942 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:44,942 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:44,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:44,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:44,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:44,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:44,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:44,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49cb08ed[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:44,943 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:44,943 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:44,944 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:44,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:44,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:44,944 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,944 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:44,944 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:44,944 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:44,945 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:44,945 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:44,945 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:44,945 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:44,946 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:44,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:44,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:44,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:44,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:44,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:44,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:44,952 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/205c1805d57d43dca4d0936321ac0af2 2023-04-08 20:55:44,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,956 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/3c80dc546b57474aa05130339c354eda as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/3c80dc546b57474aa05130339c354eda 2023-04-08 20:55:44,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:44,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:44,959 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:44,959 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:44,959 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:44,959 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:44,959 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:44,960 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:44,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:44,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:44,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:44,961 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:44,961 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:44,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:44,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:44,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:44,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:44,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:44,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:44,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:44,963 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,963 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/3c80dc546b57474aa05130339c354eda, entries=100, sequenceid=14, filesize=18.4 K 2023-04-08 20:55:44,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:44,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:44,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,964 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/5f043b692e3746d5a41bfd2c35c6d92d as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/5f043b692e3746d5a41bfd2c35c6d92d 2023-04-08 20:55:44,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:44,964 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:44,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,967 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,967 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:44,967 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:44,967 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:44,967 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:44,967 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:44,967 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:44,967 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:44,967 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:44,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:44,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:44,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:44,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:44,967 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:44,967 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:44,967 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:44,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:44,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:44,972 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/5f043b692e3746d5a41bfd2c35c6d92d, entries=100, sequenceid=14, filesize=9.3 K 2023-04-08 20:55:44,973 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/205c1805d57d43dca4d0936321ac0af2 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/205c1805d57d43dca4d0936321ac0af2 2023-04-08 20:55:44,978 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/205c1805d57d43dca4d0936321ac0af2, entries=100, sequenceid=14, filesize=9.3 K 2023-04-08 20:55:44,979 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 06f3cb8ce0404e826c51a372e2c9bfbf in 266ms, sequenceid=14, compaction requested=false 2023-04-08 20:55:44,979 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:55:45,068 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:45,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:45,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:45,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:55:45,081 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 06f3cb8ce0404e826c51a372e2c9bfbf 3/3 column families, dataSize=147.60 KB heapSize=415.45 KB 2023-04-08 20:55:45,083 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=A 2023-04-08 20:55:45,083 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-08 20:55:45,083 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=B 2023-04-08 20:55:45,083 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-08 20:55:45,083 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=C 2023-04-08 20:55:45,083 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-08 20:55:45,092 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:45,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:45,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41694 deadline: 1680987405087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41702 deadline: 1680987405087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:45,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41716 deadline: 1680987405087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,098 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:45,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41660 deadline: 1680987405092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,098 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:45,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41688 deadline: 1680987405094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:45,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:45,106 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:45,106 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:45,106 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:45,106 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:45,107 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,107 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:45,109 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:45,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:45,109 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:45,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:45,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,110 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,110 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:45,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:45,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:45,110 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:45,114 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:45,114 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:45,114 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:45,114 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:45,114 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:45,130 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:45,130 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:45,130 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:45,130 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:55:45,130 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:55:45,131 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:45,131 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:45,131 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:45,131 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,146 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:45,146 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:45,147 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,147 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:45,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:45,147 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,148 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:45,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:45,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:45,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:45,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:45,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44a8e86b[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:45,149 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:45,149 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,151 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,151 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:45,151 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:45,151 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:45,151 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,151 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:45,152 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:45,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:45,152 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:45,152 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:45,152 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:45,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:45,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:45,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:45,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:45,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:45,155 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:45,155 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:45,155 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:45,155 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:45,155 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:45,156 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,156 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:45,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,156 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:45,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:45,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:45,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:45,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:45,158 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:45,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,162 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:45,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,175 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:45,175 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:45,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:45,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:45,175 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,175 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:45,175 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:45,175 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,175 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,175 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,175 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:45,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:45,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:45,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:45,175 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:45,176 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:45,176 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:45,178 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304089728685f37474969b6e3edf3fa742526 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304089728685f37474969b6e3edf3fa742526 2023-04-08 20:55:45,193 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=52, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/ee155265376b4efeb506b8e75a8cc6ea 2023-04-08 20:55:45,202 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:45,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41702 deadline: 1680987405199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,203 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:45,204 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:45,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41716 deadline: 1680987405200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41694 deadline: 1680987405199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,204 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:45,204 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:45,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41688 deadline: 1680987405201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41660 deadline: 1680987405200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,229 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/7e5c7cb7ff9c4d95aa2a078d83da8169 2023-04-08 20:55:45,262 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/44db0d860553479cb9c80f7657e26893 2023-04-08 20:55:45,269 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/ee155265376b4efeb506b8e75a8cc6ea as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/ee155265376b4efeb506b8e75a8cc6ea 2023-04-08 20:55:45,274 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/ee155265376b4efeb506b8e75a8cc6ea, entries=150, sequenceid=52, filesize=25.1 K 2023-04-08 20:55:45,276 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/7e5c7cb7ff9c4d95aa2a078d83da8169 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/7e5c7cb7ff9c4d95aa2a078d83da8169 2023-04-08 20:55:45,276 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:45,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:45,284 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/7e5c7cb7ff9c4d95aa2a078d83da8169, entries=150, sequenceid=52, filesize=11.6 K 2023-04-08 20:55:45,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:45,287 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/44db0d860553479cb9c80f7657e26893 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/44db0d860553479cb9c80f7657e26893 2023-04-08 20:55:45,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:45,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:45,290 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:45,290 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:45,290 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:45,290 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:45,290 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,291 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:45,291 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,291 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:45,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:45,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:45,292 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,292 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:45,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:45,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:45,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:45,292 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:45,294 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:45,294 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:45,294 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:45,295 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:45,295 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:45,295 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:45,295 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:45,295 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:45,295 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:55:45,295 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:55:45,295 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:45,296 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:45,296 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:45,296 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,296 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:45,296 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:45,297 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:45,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:45,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:45,298 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:45,298 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,298 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:45,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:45,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:45,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@140c4d05[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:45,299 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:45,299 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,300 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,300 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/44db0d860553479cb9c80f7657e26893, entries=150, sequenceid=52, filesize=11.6 K 2023-04-08 20:55:45,301 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,301 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:45,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,301 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:45,301 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:45,301 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:45,301 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~436.03 KB/446496, currentSize=33.54 KB/34350 for 06f3cb8ce0404e826c51a372e2c9bfbf in 220ms, sequenceid=52, compaction requested=false 2023-04-08 20:55:45,301 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:55:45,302 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,302 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:45,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:45,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:45,302 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:45,302 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:45,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:45,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:45,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:45,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:45,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:45,307 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:45,307 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:45,307 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:45,307 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:45,307 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:45,316 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,316 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,316 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:45,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:45,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:45,317 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,318 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:45,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:45,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:45,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:45,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:45,325 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:45,325 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:45,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:45,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:45,325 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,325 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:45,325 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:45,325 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,325 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,325 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:45,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:45,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:45,325 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,326 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:45,326 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:45,325 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:45,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:45,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:45,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:55:45,408 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 06f3cb8ce0404e826c51a372e2c9bfbf 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-08 20:55:45,409 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=A 2023-04-08 20:55:45,410 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:45,410 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=B 2023-04-08 20:55:45,410 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:45,410 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=C 2023-04-08 20:55:45,410 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:45,426 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:45,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:45,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:45,430 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:45,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41694 deadline: 1680987405425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,430 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:45,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41688 deadline: 1680987405425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,430 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:45,430 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:45,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41716 deadline: 1680987405426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41702 deadline: 1680987405426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,431 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:45,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41660 deadline: 1680987405427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:45,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:45,434 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:45,434 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:45,434 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:45,434 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:45,435 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,435 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:45,436 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:45,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:45,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:45,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:45,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,437 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,437 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,437 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:45,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:45,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:45,438 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:45,440 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:45,440 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:45,440 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:45,440 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:45,440 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:45,442 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:45,442 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:45,442 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:45,442 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:55:45,442 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:55:45,442 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:45,442 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:45,442 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:45,442 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,443 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408dfa084f1d560421485fa0e46b4c7f1f2 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408dfa084f1d560421485fa0e46b4c7f1f2 2023-04-08 20:55:45,446 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:45,446 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:45,449 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,449 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:45,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:45,450 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,450 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:45,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:45,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:45,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:45,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:45,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7297f9a9[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:45,451 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:45,451 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,452 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,452 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:45,452 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:45,452 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:45,452 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,452 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:45,453 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,453 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:45,453 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:45,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:45,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:45,453 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:45,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:45,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:45,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:45,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:45,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:45,455 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:45,455 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:45,455 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:45,455 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:45,455 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:45,457 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,457 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:45,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,457 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:45,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:45,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:45,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:45,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:45,459 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:45,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,460 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:45,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,462 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:45,462 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,462 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:45,462 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:45,462 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,462 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,462 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,462 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:45,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:45,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:45,462 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:45,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:45,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:45,462 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:45,462 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:45,463 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:45,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:45,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:45,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:45,535 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:45,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41694 deadline: 1680987405531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41688 deadline: 1680987405531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:45,535 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:45,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41716 deadline: 1680987405531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41702 deadline: 1680987405531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,536 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:45,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41660 deadline: 1680987405532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,562 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:45,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:45,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:45,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:45,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:45,570 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:45,570 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:45,570 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:45,570 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:45,570 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,571 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:45,571 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:45,571 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:45,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:45,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:45,572 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,572 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:45,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:45,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:45,572 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:45,572 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:45,572 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:45,572 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:45,572 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:45,572 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:45,573 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:45,573 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:45,573 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:45,573 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:55:45,573 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:55:45,573 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:45,573 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:45,573 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:45,573 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,573 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:45,573 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:45,574 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,574 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:45,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:45,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:45,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:45,575 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,575 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:45,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:45,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:45,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@638a0b76[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:45,576 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:45,576 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,577 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,577 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:45,577 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:45,577 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,577 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:45,577 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:45,578 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,578 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:45,578 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:45,578 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:45,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:45,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:45,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:45,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:45,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:45,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:45,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:45,580 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:45,580 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:45,580 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:45,580 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:45,580 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:45,581 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:45,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:45,581 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,582 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:45,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:45,582 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:45,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:45,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:45,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,583 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:45,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,584 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,584 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:45,584 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:45,584 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:45,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:45,584 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:45,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:45,584 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:45,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:45,584 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:45,584 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,585 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,585 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,585 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:45,585 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:45,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:45,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:45,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:45,685 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:45,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:45,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:45,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:45,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:45,688 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:45,688 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:45,688 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:45,688 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:45,688 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,689 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:45,689 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:45,689 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:45,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:45,690 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,690 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:45,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:45,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:45,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:45,690 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:45,690 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:45,691 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:45,691 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:45,691 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:45,691 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:45,691 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:45,691 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:45,691 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:45,691 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:55:45,691 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:55:45,691 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:45,691 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:45,691 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:45,691 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,691 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:45,691 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:45,692 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,692 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:45,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:45,692 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,692 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:45,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:45,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:45,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:45,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:45,693 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:45,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60e25cdc[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:45,693 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,694 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,694 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:45,694 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:45,694 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:45,694 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,694 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:45,695 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,695 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:45,695 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:45,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:45,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:45,695 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:45,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:45,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:45,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:45,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:45,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:45,697 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:45,697 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:45,697 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:45,697 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:45,697 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:45,698 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,698 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:45,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:45,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,698 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:45,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:45,699 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,699 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:45,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,701 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:45,701 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:45,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:45,701 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,702 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:45,702 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:45,702 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:45,702 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:45,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:45,702 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:45,702 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:45,702 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:45,702 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,702 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:45,738 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:45,738 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:45,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41688 deadline: 1680987405737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41660 deadline: 1680987405737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,738 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:45,738 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:45,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41694 deadline: 1680987405737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41716 deadline: 1680987405738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:45,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41702 deadline: 1680987405738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,802 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:45,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:45,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:45,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:45,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:45,808 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:45,808 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:45,808 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:45,808 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:45,809 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,809 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:45,809 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:45,809 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:45,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:45,810 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,810 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:45,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:45,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:45,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:45,811 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:45,811 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:45,811 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:45,811 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:45,811 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:45,812 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:45,812 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:45,812 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:45,812 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:45,812 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:55:45,812 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:55:45,812 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:45,812 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:45,812 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:45,812 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,812 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:45,812 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:45,813 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,813 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:45,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:45,813 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,813 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:45,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:45,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:45,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:45,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:45,815 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:45,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@167041a0[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:45,815 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,815 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,818 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:45,818 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:45,818 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:45,819 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,819 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:45,819 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:45,819 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,819 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:45,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:45,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:45,819 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:45,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:45,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:45,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:45,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:45,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:45,821 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:45,821 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:45,821 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:45,821 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:45,821 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:45,822 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,822 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:45,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:45,822 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:45,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:45,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:45,823 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:45,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:45,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,826 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:45,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,841 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,841 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:45,841 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:45,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:45,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:45,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:45,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:45,841 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:45,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:45,841 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:45,842 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:45,842 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:45,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:45,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:45,842 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,842 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:45,842 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,842 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,842 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,855 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=65, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/5d1daf7ff7e342c7bb963a46f026290a 2023-04-08 20:55:45,942 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:45,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:45,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:45,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:45,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:45,946 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:45,946 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:45,946 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:45,946 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:45,947 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,947 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:45,948 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,948 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,948 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:45,954 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:45,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:45,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:45,955 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:45,955 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:45,955 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:45,955 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:45,955 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:45,955 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:45,958 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:45,958 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:45,958 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:45,958 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:55:45,958 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:55:45,958 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:45,958 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:45,958 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:45,958 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,958 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:45,958 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:45,965 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,965 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:45,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:45,966 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,966 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:45,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:45,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:45,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:45,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:45,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b67a876[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:45,968 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:45,968 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,974 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:45,975 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:45,975 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:45,975 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:45,975 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,975 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:45,983 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:45,983 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:45,983 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,983 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:45,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:45,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:45,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:45,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:45,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:45,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:45,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:45,985 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:45,985 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:45,985 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:45,985 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:45,985 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:45,991 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,991 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,991 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:45,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:45,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:45,992 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:45,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:45,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:45,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:45,994 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:45,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:45,999 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:45,999 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:45,999 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:45,999 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:45,999 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:45,999 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,000 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,000 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,000 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:45,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:46,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:45,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:46,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:45,999 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:46,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:46,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:46,000 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:46,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:46,000 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:46,040 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:46,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41660 deadline: 1680987406039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:46,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41694 deadline: 1680987406039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,042 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:46,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41702 deadline: 1680987406040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,043 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:46,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41688 deadline: 1680987406041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,043 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:46,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41716 deadline: 1680987406042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,100 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:46,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:46,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:46,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:46,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:46,103 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:46,104 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:46,104 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:46,104 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:46,104 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,104 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:46,105 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:46,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:46,105 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:46,105 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,105 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:46,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:46,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:46,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:46,106 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:46,106 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:46,106 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:46,106 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:46,106 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:46,106 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:46,106 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:46,106 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:46,106 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:46,107 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:55:46,107 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:55:46,107 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:46,107 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:46,107 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:46,107 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,107 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:46,107 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:46,108 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,108 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:46,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:46,108 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,108 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:46,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:46,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:46,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:46,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:46,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:46,110 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:46,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6885c9d7[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:46,110 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,110 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,110 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:46,110 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:46,111 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:46,111 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,111 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:46,111 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:46,111 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:46,111 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,111 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:46,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:46,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:46,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:46,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:46,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:46,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:46,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:46,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:46,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:46,113 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:46,114 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:46,114 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:46,114 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:46,114 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:46,115 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,115 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:46,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:46,115 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:46,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:46,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:46,116 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:46,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:46,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:46,120 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:46,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:46,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,133 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:46,133 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:46,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:46,133 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:46,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:46,133 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:46,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:46,133 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:46,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:46,133 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:46,133 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:46,134 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:46,133 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,134 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,134 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,134 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:46,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:46,234 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:46,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:46,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:46,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:46,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:46,237 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:46,237 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:46,237 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:46,237 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:46,237 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,237 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:46,238 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,238 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:46,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:46,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:46,238 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,238 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:46,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:46,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:46,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:46,239 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:46,239 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:46,239 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:46,239 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:46,239 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:46,239 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:46,239 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:46,239 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:46,239 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:46,240 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:55:46,240 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:55:46,240 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:46,240 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:46,240 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:46,240 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,240 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:46,240 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:46,241 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,241 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,241 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:46,242 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:46,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:46,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:46,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:46,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:46,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:46,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:46,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a737ca7[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:46,245 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:46,245 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,249 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,250 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:46,250 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:46,250 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:46,250 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,250 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:46,257 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:46,257 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,257 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:46,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:46,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:46,257 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:46,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:46,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:46,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:46,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:46,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:46,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:46,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:46,259 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:46,259 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:46,259 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:46,259 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:46,259 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:46,265 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=65 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/d604e75322fd4d40ba25ac3442a6d840 2023-04-08 20:55:46,266 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,266 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:46,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:46,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,266 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:46,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:46,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:46,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:46,270 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:46,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:46,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:46,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,272 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:46,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:46,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,275 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:46,275 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,275 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:46,275 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:46,275 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,275 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,275 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,275 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,275 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:46,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:46,275 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:46,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:46,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:46,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:46,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:46,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:46,276 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:46,276 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:46,376 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:46,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:46,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:46,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:46,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:46,390 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:46,390 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:46,390 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:46,390 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:46,390 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,391 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:46,391 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,391 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:46,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:46,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:46,391 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,392 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:46,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:46,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:46,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:46,404 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:46,407 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:46,407 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:46,407 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:46,407 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:46,407 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:46,408 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:46,408 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:46,408 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:46,408 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:55:46,408 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:55:46,408 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:46,408 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:46,408 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:46,408 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,408 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:46,408 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:46,409 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,409 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:46,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:46,410 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,410 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:46,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:46,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:46,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:46,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:46,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:46,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e9a5308[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:46,414 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:46,414 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,416 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,416 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:46,416 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:46,416 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:46,417 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:46,417 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:46,417 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:46,422 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,423 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:46,423 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:46,423 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:46,423 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:46,423 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:46,423 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:46,423 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:46,432 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,432 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:46,432 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:46,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:46,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,433 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:46,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:46,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:46,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:46,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:46,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:46,434 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:46,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:46,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,436 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,436 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:46,436 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:46,436 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,436 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:46,436 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:46,436 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:46,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:46,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:46,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:46,436 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,436 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:46,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:46,437 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:46,437 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:46,437 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:46,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:46,537 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:46,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:46,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:46,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:46,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:46,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:46,542 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:46,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41660 deadline: 1680987406541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,542 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:46,542 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:46,542 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:46,542 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,542 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:46,543 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:46,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:46,543 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:46,543 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,543 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:46,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:46,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:46,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:46,544 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:46,544 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:46,544 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:46,544 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:46,544 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:46,546 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:46,546 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:46,546 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:46,546 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:55:46,546 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:55:46,546 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:46,546 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:46,546 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:46,546 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,546 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:46,546 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:46,547 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:46,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41702 deadline: 1680987406545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,547 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,547 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:46,547 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:46,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:46,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41716 deadline: 1680987406545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,548 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,548 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:46,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:46,548 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:46,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41688 deadline: 1680987406547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:46,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:46,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:46,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:46,549 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:46,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68d86eb5[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:46,549 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,551 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,551 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:46,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:46,551 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,552 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:46,551 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:46,552 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:46,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41694 deadline: 1680987406550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,553 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,553 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:46,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:46,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:46,553 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:46,553 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:46,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:46,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:46,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:46,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:46,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:46,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:46,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:46,556 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:46,556 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:46,556 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:46,556 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:46,556 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:46,557 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,557 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,557 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:46,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:46,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:46,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:46,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:46,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:46,558 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:46,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:46,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:46,559 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:46,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:46,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,562 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:46,562 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:46,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:46,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:46,563 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:46,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:46,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:46,563 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:46,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:46,563 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,563 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:46,563 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:46,563 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:46,563 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,563 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:46,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:46,563 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,564 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,663 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:46,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:46,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:46,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:46,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:46,666 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:46,666 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:46,667 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:46,667 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:46,667 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,667 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:46,668 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:46,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:46,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:46,668 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:46,668 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,669 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:46,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:46,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:46,669 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:46,669 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:46,669 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:46,669 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:46,669 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:46,669 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:46,670 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:46,670 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:46,670 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:46,670 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:55:46,670 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:55:46,670 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:46,670 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:46,670 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:46,670 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,670 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:46,670 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:46,672 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,672 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:46,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:46,672 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:46,672 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:46,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:46,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:46,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:46,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:46,673 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:46,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e55d908[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:46,674 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,676 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,676 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:46,676 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:46,676 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:46,676 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,676 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:46,677 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,677 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:46,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:46,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:46,677 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:46,677 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:46,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:46,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:46,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:46,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:46,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:46,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:46,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:46,682 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:46,682 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:46,682 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:46,682 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:46,682 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:46,691 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,691 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,691 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:46,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:46,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:46,692 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=65 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/458e8ca2827b4b76b5ce28a7399063e6 2023-04-08 20:55:46,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:46,692 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:46,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:46,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:46,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:46,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:46,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:46,696 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,696 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/5d1daf7ff7e342c7bb963a46f026290a as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/5d1daf7ff7e342c7bb963a46f026290a 2023-04-08 20:55:46,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:46,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,700 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:46,700 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:46,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:46,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:46,700 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:46,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:46,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:46,700 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:46,700 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:46,700 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,700 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,700 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:46,700 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,700 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:46,701 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:46,700 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:46,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:46,702 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/5d1daf7ff7e342c7bb963a46f026290a, entries=200, sequenceid=65, filesize=31.9 K 2023-04-08 20:55:46,703 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/d604e75322fd4d40ba25ac3442a6d840 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/d604e75322fd4d40ba25ac3442a6d840 2023-04-08 20:55:46,707 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/d604e75322fd4d40ba25ac3442a6d840, entries=150, sequenceid=65, filesize=11.6 K 2023-04-08 20:55:46,707 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/458e8ca2827b4b76b5ce28a7399063e6 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/458e8ca2827b4b76b5ce28a7399063e6 2023-04-08 20:55:46,711 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/458e8ca2827b4b76b5ce28a7399063e6, entries=150, sequenceid=65, filesize=11.6 K 2023-04-08 20:55:46,712 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 06f3cb8ce0404e826c51a372e2c9bfbf in 1304ms, sequenceid=65, compaction requested=true 2023-04-08 20:55:46,712 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:55:46,712 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:46,712 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:55:46,712 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:55:46,712 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:46,712 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:46,713 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77230 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:55:46,713 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): 06f3cb8ce0404e826c51a372e2c9bfbf/A is initiating minor compaction (all files) 2023-04-08 20:55:46,713 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:55:46,713 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 06f3cb8ce0404e826c51a372e2c9bfbf/A in TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:46,713 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 06f3cb8ce0404e826c51a372e2c9bfbf/B is initiating minor compaction (all files) 2023-04-08 20:55:46,713 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/3c80dc546b57474aa05130339c354eda, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/ee155265376b4efeb506b8e75a8cc6ea, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/5d1daf7ff7e342c7bb963a46f026290a] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp, totalSize=75.4 K 2023-04-08 20:55:46,713 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 06f3cb8ce0404e826c51a372e2c9bfbf/B in TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:46,713 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/5f043b692e3746d5a41bfd2c35c6d92d, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/7e5c7cb7ff9c4d95aa2a078d83da8169, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/d604e75322fd4d40ba25ac3442a6d840] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp, totalSize=32.5 K 2023-04-08 20:55:46,713 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 3c80dc546b57474aa05130339c354eda, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680987344704 2023-04-08 20:55:46,714 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 5f043b692e3746d5a41bfd2c35c6d92d, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680987344704 2023-04-08 20:55:46,714 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting ee155265376b4efeb506b8e75a8cc6ea, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1680987344766 2023-04-08 20:55:46,714 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 7e5c7cb7ff9c4d95aa2a078d83da8169, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1680987344766 2023-04-08 20:55:46,714 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 5d1daf7ff7e342c7bb963a46f026290a, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1680987345406 2023-04-08 20:55:46,714 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting d604e75322fd4d40ba25ac3442a6d840, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1680987345406 2023-04-08 20:55:46,728 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 06f3cb8ce0404e826c51a372e2c9bfbf#B#compaction#136 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:46,747 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 06f3cb8ce0404e826c51a372e2c9bfbf#A#compaction#135 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:46,801 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:46,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:46,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:46,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:46,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:46,804 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:46,804 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:46,804 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:46,804 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:46,804 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,804 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:46,805 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,805 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:46,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:46,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:46,805 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:46,805 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:46,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:46,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:46,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:46,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:46,806 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:46,807 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:46,807 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:46,807 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:46,807 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:46,807 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:46,807 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:46,807 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:46,807 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:46,807 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:55:46,807 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 06f3cb8ce0404e826c51a372e2c9bfbf 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-08 20:55:46,808 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=A 2023-04-08 20:55:46,808 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:46,808 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=B 2023-04-08 20:55:46,808 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:46,808 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=C 2023-04-08 20:55:46,808 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:46,869 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040801b2e4d436f643279f975560d655e160 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040801b2e4d436f643279f975560d655e160 2023-04-08 20:55:47,002 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-08 20:55:47,160 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/3d0d5f5c72d94f948d54abda75da8b56 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/3d0d5f5c72d94f948d54abda75da8b56 2023-04-08 20:55:47,165 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 06f3cb8ce0404e826c51a372e2c9bfbf/B of 06f3cb8ce0404e826c51a372e2c9bfbf into 3d0d5f5c72d94f948d54abda75da8b56(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:47,166 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:55:47,166 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf., storeName=06f3cb8ce0404e826c51a372e2c9bfbf/B, priority=13, startTime=1680987346712; duration=0sec 2023-04-08 20:55:47,166 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:47,166 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:55:47,168 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:55:47,168 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 06f3cb8ce0404e826c51a372e2c9bfbf/C is initiating minor compaction (all files) 2023-04-08 20:55:47,168 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 06f3cb8ce0404e826c51a372e2c9bfbf/C in TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:47,169 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/205c1805d57d43dca4d0936321ac0af2, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/44db0d860553479cb9c80f7657e26893, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/458e8ca2827b4b76b5ce28a7399063e6] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp, totalSize=32.5 K 2023-04-08 20:55:47,169 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 205c1805d57d43dca4d0936321ac0af2, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680987344704 2023-04-08 20:55:47,169 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 44db0d860553479cb9c80f7657e26893, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1680987344766 2023-04-08 20:55:47,170 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 458e8ca2827b4b76b5ce28a7399063e6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1680987345406 2023-04-08 20:55:47,175 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 06f3cb8ce0404e826c51a372e2c9bfbf#C#compaction#138 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:47,185 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/3aa68c08ba2d42819d5d6a628be27587 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/3aa68c08ba2d42819d5d6a628be27587 2023-04-08 20:55:47,190 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/1aa088522a9b4a9b8ee41e6991f99bc0 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/1aa088522a9b4a9b8ee41e6991f99bc0 2023-04-08 20:55:47,192 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 06f3cb8ce0404e826c51a372e2c9bfbf/C of 06f3cb8ce0404e826c51a372e2c9bfbf into 3aa68c08ba2d42819d5d6a628be27587(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:47,192 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:55:47,192 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf., storeName=06f3cb8ce0404e826c51a372e2c9bfbf/C, priority=13, startTime=1680987346712; duration=0sec 2023-04-08 20:55:47,192 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:47,195 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 06f3cb8ce0404e826c51a372e2c9bfbf/A of 06f3cb8ce0404e826c51a372e2c9bfbf into 1aa088522a9b4a9b8ee41e6991f99bc0(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:47,195 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:55:47,195 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf., storeName=06f3cb8ce0404e826c51a372e2c9bfbf/A, priority=13, startTime=1680987346712; duration=0sec 2023-04-08 20:55:47,195 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:47,291 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=84, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/8fa6b0e2151c4c7083391d5a41bce683 2023-04-08 20:55:47,311 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/3aa6663032f04c82bbac2b5af2e55ff1 2023-04-08 20:55:47,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:55:47,547 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:55:47,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:47,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41660 deadline: 1680987407558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,562 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:47,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41702 deadline: 1680987407559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,564 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:47,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41716 deadline: 1680987407562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,564 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:47,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41688 deadline: 1680987407562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,569 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:47,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41694 deadline: 1680987407569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:47,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41660 deadline: 1680987407663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:47,665 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:47,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41716 deadline: 1680987407665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41688 deadline: 1680987407665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,672 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:47,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41702 deadline: 1680987407672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,740 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/46c918bff8a94883a0e2ef988d39c719 2023-04-08 20:55:47,744 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/8fa6b0e2151c4c7083391d5a41bce683 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/8fa6b0e2151c4c7083391d5a41bce683 2023-04-08 20:55:47,748 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/8fa6b0e2151c4c7083391d5a41bce683, entries=150, sequenceid=84, filesize=25.1 K 2023-04-08 20:55:47,748 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/3aa6663032f04c82bbac2b5af2e55ff1 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/3aa6663032f04c82bbac2b5af2e55ff1 2023-04-08 20:55:47,752 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/3aa6663032f04c82bbac2b5af2e55ff1, entries=150, sequenceid=84, filesize=11.6 K 2023-04-08 20:55:47,753 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/46c918bff8a94883a0e2ef988d39c719 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/46c918bff8a94883a0e2ef988d39c719 2023-04-08 20:55:47,758 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/46c918bff8a94883a0e2ef988d39c719, entries=150, sequenceid=84, filesize=11.6 K 2023-04-08 20:55:47,759 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 06f3cb8ce0404e826c51a372e2c9bfbf in 952ms, sequenceid=84, compaction requested=false 2023-04-08 20:55:47,759 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:55:47,759 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:47,759 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:47,759 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:47,759 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,759 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:47,759 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:47,761 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:47,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:47,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:47,761 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:47,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:47,761 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:47,762 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:47,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:47,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:47,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:47,762 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:47,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fc20e21[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:47,762 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:47,763 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:47,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:47,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:47,763 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,763 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:47,763 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:47,763 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:47,763 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:47,764 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:47,764 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,764 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:47,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:47,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:47,764 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:47,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:47,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:47,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:47,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:47,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:47,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:47,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:47,775 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:47,775 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:47,775 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:47,775 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:47,775 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:47,790 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:47,791 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:47,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:47,791 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:47,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:47,791 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:47,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:47,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:47,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:47,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:47,793 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,796 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:47,796 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:47,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:47,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:47,796 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:47,796 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:47,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:47,796 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:47,796 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:47,797 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,797 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:47,797 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:47,797 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:47,796 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:47,797 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:47,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:47,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:47,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:47,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:55:47,869 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 06f3cb8ce0404e826c51a372e2c9bfbf 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-08 20:55:47,870 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=A 2023-04-08 20:55:47,871 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:47,871 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=B 2023-04-08 20:55:47,871 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:47,871 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=C 2023-04-08 20:55:47,871 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:47,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:47,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41716 deadline: 1680987407880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:47,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41702 deadline: 1680987407881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:47,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41688 deadline: 1680987407882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,886 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:47,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41660 deadline: 1680987407883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,897 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:47,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:47,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:47,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:47,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:47,900 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:47,900 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:47,900 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:47,900 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:47,900 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:47,900 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:47,901 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:47,901 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:47,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:47,901 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,901 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:47,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:47,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:47,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:47,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:47,902 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:47,902 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:47,902 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:47,902 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:47,902 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:47,902 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:47,902 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:47,902 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:47,902 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:47,902 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:55:47,902 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:55:47,902 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:47,902 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:47,902 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:47,902 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,903 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:47,903 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:47,903 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,903 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:47,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:47,904 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:47,904 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:47,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:47,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:47,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:47,905 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:47,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@731c1ac2[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:47,905 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:47,906 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:47,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:47,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:47,906 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:47,906 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,906 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:47,906 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:47,906 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:47,907 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,907 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:47,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:47,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:47,907 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:47,907 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:47,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:47,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:47,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:47,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:47,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:47,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:47,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:47,908 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:47,908 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:47,908 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:47,908 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:47,908 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:47,909 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:47,909 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:47,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:47,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:47,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:47,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:47,909 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:47,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:47,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:47,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:47,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:47,910 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:47,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:47,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:47,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:47,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,911 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:47,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:47,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,912 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,912 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:47,912 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:47,912 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:47,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:47,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:47,913 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,913 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:47,913 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:47,913 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:47,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:47,913 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:47,913 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:47,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:47,913 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:47,913 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:47,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:47,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:47,987 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:47,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41702 deadline: 1680987407985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,988 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:47,988 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:47,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41688 deadline: 1680987407986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41716 deadline: 1680987407985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:47,988 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:47,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41660 deadline: 1680987407986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,013 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:48,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:48,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:48,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:48,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:48,016 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:48,016 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:48,016 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:48,016 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:48,016 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:48,016 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:48,017 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,017 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:48,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:48,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:48,017 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,017 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:48,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:48,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:48,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:48,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:48,018 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:48,018 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:48,018 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:48,018 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:48,018 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:48,018 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:48,018 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:48,019 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:48,019 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:48,019 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:55:48,019 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:55:48,019 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:48,019 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:48,019 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:48,019 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,019 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:48,019 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:48,020 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:48,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:48,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:48,020 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:48,020 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:48,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:48,020 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:48,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:48,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:48,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:48,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@581a6fd2[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:48,021 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:48,021 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:48,022 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:48,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:48,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:48,022 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:48,022 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:48,022 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:48,022 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,022 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:48,023 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:48,023 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,023 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:48,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:48,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:48,023 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:48,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:48,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:48,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:48,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:48,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:48,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:48,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:48,025 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:48,025 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:48,025 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:48,025 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:48,025 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:48,026 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:48,026 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:48,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:48,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:48,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:48,026 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:48,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:48,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:48,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:48,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:48,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:48,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:48,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:48,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:48,027 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:48,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:48,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:48,028 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,030 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:48,030 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:48,030 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:48,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:48,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:48,030 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:48,030 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:48,030 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:48,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:48,030 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,030 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:48,030 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:48,030 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:48,030 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:48,030 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:48,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:48,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:48,131 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:48,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:48,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:48,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:48,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:48,137 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:48,137 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:48,137 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:48,137 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:48,137 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:48,137 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:48,138 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,138 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:48,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:48,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:48,139 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,139 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:48,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:48,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:48,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:48,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:48,140 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:48,146 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:48,146 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:48,146 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:48,146 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:48,146 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:48,146 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:48,146 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:48,146 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:48,147 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:55:48,147 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:55:48,147 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:48,147 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:48,147 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:48,147 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,147 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:48,147 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:48,152 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:48,152 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:48,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:48,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:48,153 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:48,153 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:48,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:48,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:48,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:48,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:48,154 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:48,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58076040[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:48,154 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:48,154 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:48,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:48,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:48,154 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:48,154 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:48,154 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:48,155 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,155 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:48,155 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:48,155 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:48,155 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,155 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:48,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:48,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:48,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:48,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:48,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:48,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:48,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:48,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:48,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:48,159 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:48,159 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:48,159 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:48,159 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:48,159 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:48,160 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:48,160 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:48,160 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:48,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:48,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:48,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:48,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:48,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:48,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:48,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:48,161 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:48,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:48,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:48,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:48,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:48,164 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:48,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:48,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,165 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:48,165 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:48,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:48,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:48,166 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,166 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:48,166 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:48,166 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,166 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:48,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:48,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:48,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:48,166 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:48,166 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:48,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:48,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:48,166 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:48,167 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:48,167 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:48,191 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:48,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41660 deadline: 1680987408190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,191 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:48,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41688 deadline: 1680987408190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,191 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:48,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41716 deadline: 1680987408190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,192 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:48,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41702 deadline: 1680987408190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,267 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:48,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:48,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:48,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:48,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:48,282 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:48,282 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:48,282 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:48,282 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:48,282 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:48,283 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:48,284 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:48,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:48,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:48,284 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:48,285 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,285 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:48,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:48,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:48,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:48,285 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:48,290 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:48,290 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:48,290 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:48,290 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:48,290 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:48,290 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:48,290 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:48,291 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:48,291 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:55:48,291 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:55:48,291 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:48,291 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:48,291 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:48,291 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,291 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:48,291 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:48,295 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408303335c41c164086bf55a70f3320bc46 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408303335c41c164086bf55a70f3320bc46 2023-04-08 20:55:48,298 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:48,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:48,298 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:48,298 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:48,298 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:48,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:48,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:48,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:48,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:48,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:48,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@678bce67[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:48,300 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:48,300 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:48,301 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:48,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:48,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:48,301 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:48,301 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:48,301 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:48,302 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,302 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:48,302 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:48,302 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:48,302 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:48,302 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:48,302 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:48,302 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:48,302 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:48,302 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:48,303 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:48,304 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:48,304 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:48,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:48,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:48,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:48,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:48,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:48,304 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:48,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:48,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:48,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:48,305 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:48,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:48,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:48,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:48,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:48,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:48,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,307 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,308 WARN [DataStreamer for file /user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/2a795156e07241f79d5f03b5453e8a47] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-08 20:55:48,308 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=103, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/2a795156e07241f79d5f03b5453e8a47 2023-04-08 20:55:48,309 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:48,309 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:48,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:48,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:48,309 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,309 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:48,309 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:48,309 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,309 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:48,309 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:48,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:48,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:48,309 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:48,310 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:48,310 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:48,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:48,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:48,310 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:48,330 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=103 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/4a266c949449432d87425df1d5213b3c 2023-04-08 20:55:48,354 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=103 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/398b7bfd810e45ea8452569d63656073 2023-04-08 20:55:48,362 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/2a795156e07241f79d5f03b5453e8a47 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/2a795156e07241f79d5f03b5453e8a47 2023-04-08 20:55:48,367 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/2a795156e07241f79d5f03b5453e8a47, entries=200, sequenceid=103, filesize=31.9 K 2023-04-08 20:55:48,369 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/4a266c949449432d87425df1d5213b3c as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/4a266c949449432d87425df1d5213b3c 2023-04-08 20:55:48,375 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/4a266c949449432d87425df1d5213b3c, entries=150, sequenceid=103, filesize=11.6 K 2023-04-08 20:55:48,377 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/398b7bfd810e45ea8452569d63656073 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/398b7bfd810e45ea8452569d63656073 2023-04-08 20:55:48,383 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/398b7bfd810e45ea8452569d63656073, entries=150, sequenceid=103, filesize=11.6 K 2023-04-08 20:55:48,384 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 06f3cb8ce0404e826c51a372e2c9bfbf in 515ms, sequenceid=103, compaction requested=true 2023-04-08 20:55:48,384 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:55:48,384 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:48,384 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:55:48,384 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:55:48,384 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:48,384 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:48,385 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:55:48,386 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 06f3cb8ce0404e826c51a372e2c9bfbf/B is initiating minor compaction (all files) 2023-04-08 20:55:48,386 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 06f3cb8ce0404e826c51a372e2c9bfbf/B in TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:48,386 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/3d0d5f5c72d94f948d54abda75da8b56, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/3aa6663032f04c82bbac2b5af2e55ff1, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/4a266c949449432d87425df1d5213b3c] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp, totalSize=34.9 K 2023-04-08 20:55:48,386 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84281 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:55:48,386 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 3d0d5f5c72d94f948d54abda75da8b56, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1680987345406 2023-04-08 20:55:48,386 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): 06f3cb8ce0404e826c51a372e2c9bfbf/A is initiating minor compaction (all files) 2023-04-08 20:55:48,386 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 06f3cb8ce0404e826c51a372e2c9bfbf/A in TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:48,386 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/1aa088522a9b4a9b8ee41e6991f99bc0, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/8fa6b0e2151c4c7083391d5a41bce683, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/2a795156e07241f79d5f03b5453e8a47] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp, totalSize=82.3 K 2023-04-08 20:55:48,387 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 3aa6663032f04c82bbac2b5af2e55ff1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1680987345423 2023-04-08 20:55:48,387 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 1aa088522a9b4a9b8ee41e6991f99bc0, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1680987345406 2023-04-08 20:55:48,387 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 4a266c949449432d87425df1d5213b3c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1680987347558 2023-04-08 20:55:48,388 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 8fa6b0e2151c4c7083391d5a41bce683, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1680987345423 2023-04-08 20:55:48,388 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 2a795156e07241f79d5f03b5453e8a47, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1680987347552 2023-04-08 20:55:48,397 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 06f3cb8ce0404e826c51a372e2c9bfbf#B#compaction#144 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:48,400 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 06f3cb8ce0404e826c51a372e2c9bfbf#A#compaction#145 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:48,410 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:48,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:48,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:48,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:48,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:48,424 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:48,424 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:48,424 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:48,424 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:48,425 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:48,425 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:48,426 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:48,426 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:48,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:48,426 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:48,426 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:48,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:48,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:48,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:48,427 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:48,432 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:48,433 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:48,433 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:48,433 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:48,433 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:48,433 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:48,433 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:48,433 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:48,433 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:55:48,433 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 06f3cb8ce0404e826c51a372e2c9bfbf 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-08 20:55:48,434 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=A 2023-04-08 20:55:48,434 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:48,434 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=B 2023-04-08 20:55:48,434 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:48,434 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=C 2023-04-08 20:55:48,434 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:48,484 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408fbd97c5eceac4b07ae15c42566393cca to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408fbd97c5eceac4b07ae15c42566393cca 2023-04-08 20:55:48,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:55:48,496 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:55:48,521 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:48,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41660 deadline: 1680987408517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:48,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41702 deadline: 1680987408519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:48,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41688 deadline: 1680987408520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:48,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41716 deadline: 1680987408521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,625 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:48,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41660 deadline: 1680987408622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,626 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:48,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41702 deadline: 1680987408623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,626 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:48,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41688 deadline: 1680987408624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:48,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41716 deadline: 1680987408634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,828 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:48,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41660 deadline: 1680987408827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,833 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/413c7d407f534a6babd50042971f7427 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/413c7d407f534a6babd50042971f7427 2023-04-08 20:55:48,834 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:48,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41702 deadline: 1680987408832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:48,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41688 deadline: 1680987408836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,837 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 06f3cb8ce0404e826c51a372e2c9bfbf/B of 06f3cb8ce0404e826c51a372e2c9bfbf into 413c7d407f534a6babd50042971f7427(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:48,837 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:55:48,838 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf., storeName=06f3cb8ce0404e826c51a372e2c9bfbf/B, priority=13, startTime=1680987348384; duration=0sec 2023-04-08 20:55:48,838 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:48,838 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:55:48,839 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:55:48,839 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 06f3cb8ce0404e826c51a372e2c9bfbf/C is initiating minor compaction (all files) 2023-04-08 20:55:48,839 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 06f3cb8ce0404e826c51a372e2c9bfbf/C in TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:48,839 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/3aa68c08ba2d42819d5d6a628be27587, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/46c918bff8a94883a0e2ef988d39c719, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/398b7bfd810e45ea8452569d63656073] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp, totalSize=34.9 K 2023-04-08 20:55:48,839 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 3aa68c08ba2d42819d5d6a628be27587, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1680987345406 2023-04-08 20:55:48,840 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 46c918bff8a94883a0e2ef988d39c719, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1680987345423 2023-04-08 20:55:48,840 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 398b7bfd810e45ea8452569d63656073, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1680987347558 2023-04-08 20:55:48,844 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:48,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41716 deadline: 1680987408841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:48,847 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 06f3cb8ce0404e826c51a372e2c9bfbf#C#compaction#147 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:48,896 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=119, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/9495de42f4874f9582274e9540fd3283 2023-04-08 20:55:49,129 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:49,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41660 deadline: 1680987409129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:49,135 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:49,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41702 deadline: 1680987409135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:49,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:49,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41688 deadline: 1680987409139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:49,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:49,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41716 deadline: 1680987409147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:49,255 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/b2f920cf84f24135a0eec58cc131ff29 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/b2f920cf84f24135a0eec58cc131ff29 2023-04-08 20:55:49,260 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 06f3cb8ce0404e826c51a372e2c9bfbf/C of 06f3cb8ce0404e826c51a372e2c9bfbf into b2f920cf84f24135a0eec58cc131ff29(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:49,261 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:55:49,261 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf., storeName=06f3cb8ce0404e826c51a372e2c9bfbf/C, priority=13, startTime=1680987348384; duration=0sec 2023-04-08 20:55:49,261 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:49,309 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=119 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/ae54f2a3192f490db7cb70258c4cda70 2023-04-08 20:55:49,330 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=119 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/fe2b9554e18b4d6ca180c3af45a12a91 2023-04-08 20:55:49,334 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/9495de42f4874f9582274e9540fd3283 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/9495de42f4874f9582274e9540fd3283 2023-04-08 20:55:49,337 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/9495de42f4874f9582274e9540fd3283, entries=100, sequenceid=119, filesize=18.4 K 2023-04-08 20:55:49,354 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/ae54f2a3192f490db7cb70258c4cda70 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/ae54f2a3192f490db7cb70258c4cda70 2023-04-08 20:55:49,359 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/ae54f2a3192f490db7cb70258c4cda70, entries=100, sequenceid=119, filesize=9.3 K 2023-04-08 20:55:49,360 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/fe2b9554e18b4d6ca180c3af45a12a91 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/fe2b9554e18b4d6ca180c3af45a12a91 2023-04-08 20:55:49,363 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/fe2b9554e18b4d6ca180c3af45a12a91, entries=100, sequenceid=119, filesize=9.3 K 2023-04-08 20:55:49,364 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 06f3cb8ce0404e826c51a372e2c9bfbf in 931ms, sequenceid=119, compaction requested=false 2023-04-08 20:55:49,364 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:55:49,364 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:49,364 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:49,364 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:49,364 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:49,368 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:49,368 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:49,369 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:49,369 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:49,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:49,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:49,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:49,369 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:49,369 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:49,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:49,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:49,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:49,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:49,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:49,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:49,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7452125e[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:49,371 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:49,371 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:49,371 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:49,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:49,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:49,371 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:49,372 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:49,372 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:49,372 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:49,372 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:49,372 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:49,372 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:49,372 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:49,373 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:49,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:49,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:49,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:49,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:49,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:49,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:49,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:49,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:49,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:49,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:49,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:49,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:49,375 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:49,375 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:49,375 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:49,375 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:49,375 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:49,376 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:49,376 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:49,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:49,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:49,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:49,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:49,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:49,377 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:49,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:49,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:49,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:49,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:49,378 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:49,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:49,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:49,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:49,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:49,379 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:49,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:49,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:49,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:49,381 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:49,381 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:49,381 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:49,381 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:49,381 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:49,381 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:49,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:49,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:49,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:49,381 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:49,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:49,381 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:49,381 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:49,381 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:49,381 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:49,381 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:49,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:49,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:49,481 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:49,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:49,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:49,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:49,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:49,484 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:49,484 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:49,484 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:49,484 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:49,484 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:49,485 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:49,485 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:49,485 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:49,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:49,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:49,486 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:49,486 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:49,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:49,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:49,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:49,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:49,486 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:49,486 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:49,486 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:49,487 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:49,487 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:49,487 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:49,487 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:49,487 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:49,487 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:49,487 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:55:49,487 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 06f3cb8ce0404e826c51a372e2c9bfbf 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-08 20:55:49,487 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=A 2023-04-08 20:55:49,487 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:49,487 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=B 2023-04-08 20:55:49,487 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:49,487 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=C 2023-04-08 20:55:49,487 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:49,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:55:49,587 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:55:49,624 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:49,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41694 deadline: 1680987409623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:49,634 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:49,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41660 deadline: 1680987409633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:49,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:49,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41702 deadline: 1680987409640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:49,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:49,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41688 deadline: 1680987409641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:49,650 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:49,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41716 deadline: 1680987409649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:49,655 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/ad3bd7f330584292bbaf3d629f0ba73c as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/ad3bd7f330584292bbaf3d629f0ba73c 2023-04-08 20:55:49,661 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 06f3cb8ce0404e826c51a372e2c9bfbf/A of 06f3cb8ce0404e826c51a372e2c9bfbf into ad3bd7f330584292bbaf3d629f0ba73c(size=25.3 K), total size for store is 43.7 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-08 20:55:49,661 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:55:49,662 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf., storeName=06f3cb8ce0404e826c51a372e2c9bfbf/A, priority=13, startTime=1680987348384; duration=1sec 2023-04-08 20:55:49,665 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:49,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:49,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41694 deadline: 1680987409725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:49,902 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040869863ac5d402407b86920e7e361fb288 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040869863ac5d402407b86920e7e361fb288 2023-04-08 20:55:49,929 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:49,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41694 deadline: 1680987409928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:50,232 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:50,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41694 deadline: 1680987410230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:50,308 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=137, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/2344f51dbb5a4b638bc3098e191b9909 2023-04-08 20:55:50,319 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=137 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/847561cdaf9142faadaa3fa9708abf37 2023-04-08 20:55:50,645 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:50,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41702 deadline: 1680987410643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:50,646 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:50,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41660 deadline: 1680987410644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:50,652 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:50,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41688 deadline: 1680987410651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:50,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:50,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41716 deadline: 1680987410653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:50,729 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=137 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/d411662c97f140608a40c07b815145b7 2023-04-08 20:55:50,737 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/2344f51dbb5a4b638bc3098e191b9909 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/2344f51dbb5a4b638bc3098e191b9909 2023-04-08 20:55:50,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:50,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41694 deadline: 1680987410737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:50,740 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/2344f51dbb5a4b638bc3098e191b9909, entries=150, sequenceid=137, filesize=25.3 K 2023-04-08 20:55:50,742 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/847561cdaf9142faadaa3fa9708abf37 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/847561cdaf9142faadaa3fa9708abf37 2023-04-08 20:55:50,746 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/847561cdaf9142faadaa3fa9708abf37, entries=150, sequenceid=137, filesize=11.8 K 2023-04-08 20:55:50,747 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/d411662c97f140608a40c07b815145b7 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/d411662c97f140608a40c07b815145b7 2023-04-08 20:55:50,760 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/d411662c97f140608a40c07b815145b7, entries=150, sequenceid=137, filesize=11.8 K 2023-04-08 20:55:50,761 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 06f3cb8ce0404e826c51a372e2c9bfbf in 1274ms, sequenceid=137, compaction requested=true 2023-04-08 20:55:50,761 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:55:50,761 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:50,761 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:50,761 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:50,761 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:50,761 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:50,761 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:50,762 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:50,762 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:50,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:50,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:50,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:50,763 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:50,763 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:50,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:50,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:50,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:50,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:50,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:50,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:50,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fbd97f9[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:50,766 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:50,766 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:50,767 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:50,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:50,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:50,768 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:50,768 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:50,768 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:50,768 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:50,768 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:50,769 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:50,769 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:50,769 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:50,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:50,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:50,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:50,769 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:50,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:50,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:50,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:50,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:50,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:50,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:50,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:50,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:50,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:50,772 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:50,772 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:50,772 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:50,772 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:50,772 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:50,777 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:50,777 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:50,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:50,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:50,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:50,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:50,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:50,777 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:50,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:50,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:50,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:50,778 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:50,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:50,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:50,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:50,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:50,779 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:50,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:50,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:50,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:50,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:50,782 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:50,782 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:50,782 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:50,782 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:50,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:50,782 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:50,782 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:50,782 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:50,782 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:50,782 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:50,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:50,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:50,782 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:50,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:50,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:50,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:50,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:50,785 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:50,785 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:50,885 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:50,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:50,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:50,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:50,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:50,888 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:50,888 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:50,888 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:50,888 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:50,888 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:50,888 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:50,888 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:50,889 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:50,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:50,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:50,889 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:50,889 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:50,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:50,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:50,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:50,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:50,889 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:50,890 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:50,890 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:50,890 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:50,890 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:50,890 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:50,890 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:50,890 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:50,890 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:50,890 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:55:50,890 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 06f3cb8ce0404e826c51a372e2c9bfbf 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-08 20:55:50,890 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=A 2023-04-08 20:55:50,890 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:50,890 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=B 2023-04-08 20:55:50,890 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:50,890 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=C 2023-04-08 20:55:50,890 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:50,902 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304086dc6a579164346e397ed12047296137a to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304086dc6a579164346e397ed12047296137a 2023-04-08 20:55:51,308 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=154, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/a761fe9ad79a430f837a670750aa2bad 2023-04-08 20:55:51,718 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/4cc3f4af84614958b75109bc42c42070 2023-04-08 20:55:51,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:55:51,747 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:55:51,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:51,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41694 deadline: 1680987411779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:51,883 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:51,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41694 deadline: 1680987411882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:52,085 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:52,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41694 deadline: 1680987412084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:52,128 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/9248ccf6efa242689421db2bd960dd9a 2023-04-08 20:55:52,133 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/a761fe9ad79a430f837a670750aa2bad as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/a761fe9ad79a430f837a670750aa2bad 2023-04-08 20:55:52,136 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/a761fe9ad79a430f837a670750aa2bad, entries=150, sequenceid=154, filesize=25.3 K 2023-04-08 20:55:52,137 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/4cc3f4af84614958b75109bc42c42070 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/4cc3f4af84614958b75109bc42c42070 2023-04-08 20:55:52,140 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/4cc3f4af84614958b75109bc42c42070, entries=150, sequenceid=154, filesize=11.8 K 2023-04-08 20:55:52,141 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/9248ccf6efa242689421db2bd960dd9a as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/9248ccf6efa242689421db2bd960dd9a 2023-04-08 20:55:52,144 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/9248ccf6efa242689421db2bd960dd9a, entries=150, sequenceid=154, filesize=11.8 K 2023-04-08 20:55:52,145 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 06f3cb8ce0404e826c51a372e2c9bfbf in 1255ms, sequenceid=154, compaction requested=true 2023-04-08 20:55:52,145 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:55:52,145 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:52,145 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:52,145 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:52,145 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:52,145 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:52,145 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:52,147 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:52,147 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:52,148 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:52,148 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:52,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:52,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:52,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:52,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:52,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:52,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:52,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c5bfd2e[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:52,149 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:52,149 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:52,149 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:52,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:52,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:52,149 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:52,149 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:52,149 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:52,149 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:52,149 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:52,150 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:52,150 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:52,150 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:52,154 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:52,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:52,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:52,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:52,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:52,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:52,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:52,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:52,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:52,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:52,156 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:52,157 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:52,157 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:52,157 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:52,157 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:52,158 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:52,158 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:52,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:52,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:52,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:52,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:52,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:52,158 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:52,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:52,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:52,158 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:52,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:52,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:52,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:52,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:52,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:52,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:52,160 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:52,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:52,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:52,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:52,162 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:52,162 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:52,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:52,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:52,162 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:52,162 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:52,162 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:52,162 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:52,162 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:52,162 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:52,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:52,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:52,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:52,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:52,162 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:52,162 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:52,163 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:52,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:52,163 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:52,263 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:52,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:52,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:52,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:52,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:52,266 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:52,266 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:52,266 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:52,266 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:52,267 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:52,267 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:52,267 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:52,267 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:52,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:52,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:52,268 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:52,268 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:52,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:52,268 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:52,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:52,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:52,268 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:52,268 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:52,268 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:52,268 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:52,268 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:52,269 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:52,269 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:52,269 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:52,269 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:52,269 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:55:52,269 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 06f3cb8ce0404e826c51a372e2c9bfbf 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-08 20:55:52,269 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=A 2023-04-08 20:55:52,269 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:52,269 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=B 2023-04-08 20:55:52,269 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:52,269 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=C 2023-04-08 20:55:52,270 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:52,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:55:52,388 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:55:52,452 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:52,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41694 deadline: 1680987412448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:52,558 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:52,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41694 deadline: 1680987412553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:52,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:52,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41660 deadline: 1680987412647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:52,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:52,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41702 deadline: 1680987412648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:52,651 DEBUG [Thread-970] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4131 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf., hostname=jenkins-hbase20.apache.org,41091,1680987313601, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 20:55:52,651 DEBUG [Thread-976] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4132 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf., hostname=jenkins-hbase20.apache.org,41091,1680987313601, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 20:55:52,663 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:52,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41688 deadline: 1680987412662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:52,665 DEBUG [Thread-972] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4144 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf., hostname=jenkins-hbase20.apache.org,41091,1680987313601, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 20:55:52,667 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:52,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41716 deadline: 1680987412666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:52,668 DEBUG [Thread-978] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4146 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf., hostname=jenkins-hbase20.apache.org,41091,1680987313601, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 20:55:52,683 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304086b4b887829a44649a5796127e27f7f8a to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304086b4b887829a44649a5796127e27f7f8a 2023-04-08 20:55:52,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:52,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41694 deadline: 1680987412759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:53,065 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:53,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41694 deadline: 1680987413063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:53,088 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=170, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/6bbba4261e0c425392e0c0bc40c4c1f1 2023-04-08 20:55:53,102 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/7c3b41df8847405ba6bfad50853493c9 2023-04-08 20:55:53,118 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/2c529efc887447a28c97802955d6c4ea 2023-04-08 20:55:53,122 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/6bbba4261e0c425392e0c0bc40c4c1f1 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/6bbba4261e0c425392e0c0bc40c4c1f1 2023-04-08 20:55:53,126 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/6bbba4261e0c425392e0c0bc40c4c1f1, entries=150, sequenceid=170, filesize=25.3 K 2023-04-08 20:55:53,126 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/7c3b41df8847405ba6bfad50853493c9 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/7c3b41df8847405ba6bfad50853493c9 2023-04-08 20:55:53,136 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/7c3b41df8847405ba6bfad50853493c9, entries=150, sequenceid=170, filesize=11.8 K 2023-04-08 20:55:53,137 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/2c529efc887447a28c97802955d6c4ea as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/2c529efc887447a28c97802955d6c4ea 2023-04-08 20:55:53,142 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/2c529efc887447a28c97802955d6c4ea, entries=150, sequenceid=170, filesize=11.8 K 2023-04-08 20:55:53,143 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 06f3cb8ce0404e826c51a372e2c9bfbf in 874ms, sequenceid=170, compaction requested=true 2023-04-08 20:55:53,143 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:55:53,143 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:53,143 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:53,143 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:53,143 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:53,143 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:53,143 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:53,144 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:53,144 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:53,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:53,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:53,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:53,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:53,145 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:53,145 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:53,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:53,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:53,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:53,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:53,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:53,147 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:53,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@251ae034[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:53,147 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:53,148 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:53,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:53,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:53,148 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:53,148 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:53,148 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:53,148 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:53,148 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:53,149 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:53,149 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:53,149 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:53,149 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:53,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:53,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:53,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:53,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:53,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:53,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:53,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:53,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:53,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:53,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:53,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:53,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:53,151 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:53,151 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:53,151 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:53,151 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:53,151 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:53,152 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:53,152 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:53,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:53,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:53,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:53,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:53,152 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:53,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:53,152 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:53,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:53,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:53,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:53,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:53,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:53,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:53,153 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:53,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:53,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:53,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:53,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:53,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:53,155 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:53,155 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:53,155 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:53,155 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:53,155 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:53,155 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:53,155 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:53,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:53,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:53,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:53,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:53,155 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:53,155 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:53,155 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:53,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:53,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:53,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:53,156 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:53,156 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:53,256 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:53,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:53,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:53,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:53,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:53,258 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:53,258 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:53,258 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:53,259 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:53,259 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:53,259 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:53,260 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:53,260 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:53,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:53,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:53,260 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:53,260 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:53,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:53,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:53,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:53,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:53,261 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:53,261 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:53,261 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:53,261 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:53,261 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:53,261 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:53,261 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:53,261 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:53,261 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:53,261 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:55:53,261 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 06f3cb8ce0404e826c51a372e2c9bfbf 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-08 20:55:53,262 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=A 2023-04-08 20:55:53,262 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:53,262 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=B 2023-04-08 20:55:53,262 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:53,262 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=C 2023-04-08 20:55:53,262 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:53,285 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408932f1630221e4412acc5adee6c97e1b2 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408932f1630221e4412acc5adee6c97e1b2 2023-04-08 20:55:53,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:55:53,571 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:55:53,617 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:53,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41694 deadline: 1680987413615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:53,695 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=186, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/7dbb9acbed244341ab2bb8faaa27ef77 2023-04-08 20:55:53,719 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:53,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41694 deadline: 1680987413718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:53,924 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:53,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41694 deadline: 1680987413920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,105 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=186 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/adb92f0f261044799a9cb438236b1006 2023-04-08 20:55:54,144 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=186 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/4cab6128916f459c891cd3ccea997e03 2023-04-08 20:55:54,151 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/7dbb9acbed244341ab2bb8faaa27ef77 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/7dbb9acbed244341ab2bb8faaa27ef77 2023-04-08 20:55:54,156 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/7dbb9acbed244341ab2bb8faaa27ef77, entries=150, sequenceid=186, filesize=25.3 K 2023-04-08 20:55:54,157 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/adb92f0f261044799a9cb438236b1006 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/adb92f0f261044799a9cb438236b1006 2023-04-08 20:55:54,165 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/adb92f0f261044799a9cb438236b1006, entries=150, sequenceid=186, filesize=11.8 K 2023-04-08 20:55:54,166 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/4cab6128916f459c891cd3ccea997e03 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/4cab6128916f459c891cd3ccea997e03 2023-04-08 20:55:54,172 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/4cab6128916f459c891cd3ccea997e03, entries=150, sequenceid=186, filesize=11.8 K 2023-04-08 20:55:54,173 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 06f3cb8ce0404e826c51a372e2c9bfbf in 912ms, sequenceid=186, compaction requested=true 2023-04-08 20:55:54,173 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:55:54,173 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:54,173 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:54,173 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:54,173 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,174 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:54,174 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:54,175 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,175 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:54,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:54,175 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,175 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:54,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:54,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:54,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:54,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:54,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:54,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41bec62b[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:54,176 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:54,176 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,177 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,177 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:54,177 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:54,177 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:54,177 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,177 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:54,178 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:54,178 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:54,178 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:54,178 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:54,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:54,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:54,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:54,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:54,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:54,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:54,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:54,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:54,181 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:54,181 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:54,181 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:54,181 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:54,181 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:54,182 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,182 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:54,182 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:54,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:54,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:54,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:54,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:54,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,183 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:54,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:54,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:54,184 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:54,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:54,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,186 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,186 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:54,186 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:54,186 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:54,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:54,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:54,186 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:54,186 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:54,186 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:54,186 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:54,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:54,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:54,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:54,186 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,186 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,187 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:54,187 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:54,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:55:54,230 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 06f3cb8ce0404e826c51a372e2c9bfbf 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-08 20:55:54,230 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=A 2023-04-08 20:55:54,230 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:54,230 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=B 2023-04-08 20:55:54,230 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:54,230 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=C 2023-04-08 20:55:54,231 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:54,252 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408a916e664419c41319ad436c54ef4828d to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408a916e664419c41319ad436c54ef4828d 2023-04-08 20:55:54,287 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:54,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:54,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:54,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41694 deadline: 1680987414284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:54,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:54,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:54,290 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:54,290 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:54,290 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:54,290 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:54,291 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,291 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:54,291 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:54,291 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:54,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:54,292 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,292 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:54,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:54,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:54,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:54,293 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:54,293 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:54,293 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:54,293 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:54,293 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:54,293 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:54,298 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:54,298 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:54,298 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:54,298 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:55:54,298 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:55:54,298 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:54,298 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:54,298 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:54,298 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,298 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:54,298 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:54,305 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,305 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:54,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:54,306 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,306 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:54,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:54,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:54,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:54,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:54,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:54,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@522f93ba[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:54,308 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:54,308 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,310 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,310 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:54,310 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:54,310 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:54,311 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,311 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:54,313 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:54,313 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,313 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:54,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:54,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:54,313 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:54,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:54,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:54,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:54,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:54,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:54,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:54,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:54,315 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:54,315 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:54,315 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:54,315 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:54,315 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:54,316 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,316 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:54,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:54,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,316 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:54,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:54,316 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:54,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:54,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:54,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:54,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:54,317 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:54,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:54,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,320 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,320 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:54,320 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:54,320 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:54,320 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,320 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:54,320 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:54,320 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:54,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:54,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:54,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:54,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:54,320 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,321 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:54,321 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:54,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:54,321 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:54,391 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:54,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41694 deadline: 1680987414389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,421 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:54,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:54,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:54,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:54,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:54,425 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:54,425 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:54,425 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:54,425 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:54,425 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,425 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:54,426 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,426 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:54,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:54,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:54,426 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,426 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:54,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:54,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:54,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:54,427 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:54,427 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:54,427 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:54,427 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:54,427 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:54,427 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:54,427 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:54,427 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:54,427 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:54,427 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:55:54,427 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:55:54,427 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:54,428 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:54,428 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:54,428 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,428 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:54,428 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:54,429 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,429 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:54,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:54,429 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,429 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:54,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:54,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:54,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:54,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:54,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:54,430 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:54,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11deb65c[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:54,430 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,431 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,431 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:54,431 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:54,431 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:54,431 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,431 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:54,432 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:54,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:54,432 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:54,432 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:54,432 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:54,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:54,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:54,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:54,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:54,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:54,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:54,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:54,434 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:54,434 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:54,434 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:54,434 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:54,434 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:54,435 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,435 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:54,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:54,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,435 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:54,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,435 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:54,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:54,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:54,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:54,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:54,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:54,436 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:54,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:54,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,438 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:54,438 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:54,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:54,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:54,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:54,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:54,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:54,438 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,438 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:54,438 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:54,438 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:54,438 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:54,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:54,438 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,438 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:54,438 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:54,438 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,438 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,538 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:54,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:54,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:54,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:54,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:54,541 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:54,541 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:54,541 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:54,541 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:54,541 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,541 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:54,542 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:54,542 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:54,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:54,542 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,542 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:54,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:54,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:54,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:54,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:54,543 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:54,543 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:54,543 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:54,543 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:54,543 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:54,543 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:54,544 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:54,544 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:54,544 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:55:54,544 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:55:54,544 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:54,544 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:54,544 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:54,544 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,544 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:54,544 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:54,545 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:54,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:54,545 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:54,545 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,545 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:54,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:54,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:54,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:54,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:54,546 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:54,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6337ff13[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:54,546 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,547 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,547 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:54,547 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,547 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:54,547 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:54,547 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:54,548 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:54,548 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:54,548 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,548 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:54,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:54,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:54,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:54,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:54,550 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:54,550 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:54,550 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:54,550 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:54,550 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:54,551 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,551 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:54,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:54,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,551 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:54,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:54,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:54,551 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:54,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:54,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:54,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:54,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,552 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:54,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:54,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,558 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,558 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:54,558 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:54,558 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,558 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:54,558 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:54,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:54,558 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:54,558 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,558 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,558 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:54,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:54,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:54,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:54,559 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:54,559 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:54,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:54,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:54,594 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:54,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41694 deadline: 1680987414593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,659 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:54,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:54,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:54,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:54,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:54,662 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:54,662 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:54,662 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:54,662 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:54,663 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,663 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:54,663 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:54,663 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:54,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:54,664 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,664 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:54,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:54,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:54,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:54,664 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:54,674 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:54,675 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:54,675 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:54,675 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:54,675 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:54,675 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:54,675 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:54,675 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:54,676 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:55:54,676 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:55:54,676 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:54,676 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:54,676 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:54,676 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,676 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:54,676 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:54,677 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,677 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:54,677 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,677 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:54,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:54,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:54,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:54,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:54,679 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:54,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@540fe2c0[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:54,679 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,680 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,680 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:54,680 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:54,680 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,680 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:54,680 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:54,682 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,682 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:54,682 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:54,682 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:54,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:54,684 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:54,684 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:54,684 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:54,684 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:54,685 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:54,686 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,686 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:54,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:54,686 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:54,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:54,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:54,687 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:54,688 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=203, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/100bd58e18394ee2ae96f06fb8026f62 2023-04-08 20:55:54,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:54,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,689 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:54,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:54,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,694 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:54,694 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:54,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:54,694 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,694 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:54,694 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:54,694 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,694 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:54,694 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:54,694 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,694 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:54,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:54,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:54,695 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:54,695 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:54,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:54,710 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/2c6107af49e84ec0a53b302bc308eaaa 2023-04-08 20:55:54,727 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/988044634b4b4043860fd656ead62259 2023-04-08 20:55:54,731 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/100bd58e18394ee2ae96f06fb8026f62 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/100bd58e18394ee2ae96f06fb8026f62 2023-04-08 20:55:54,734 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/100bd58e18394ee2ae96f06fb8026f62, entries=150, sequenceid=203, filesize=25.3 K 2023-04-08 20:55:54,735 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/2c6107af49e84ec0a53b302bc308eaaa as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/2c6107af49e84ec0a53b302bc308eaaa 2023-04-08 20:55:54,747 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/2c6107af49e84ec0a53b302bc308eaaa, entries=150, sequenceid=203, filesize=11.8 K 2023-04-08 20:55:54,748 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/988044634b4b4043860fd656ead62259 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/988044634b4b4043860fd656ead62259 2023-04-08 20:55:54,755 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/988044634b4b4043860fd656ead62259, entries=150, sequenceid=203, filesize=11.8 K 2023-04-08 20:55:54,762 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 06f3cb8ce0404e826c51a372e2c9bfbf in 532ms, sequenceid=203, compaction requested=true 2023-04-08 20:55:54,762 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:55:54,762 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:54,762 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-08 20:55:54,762 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-08 20:55:54,762 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:54,763 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:54,767 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 174230 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-08 20:55:54,767 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): 06f3cb8ce0404e826c51a372e2c9bfbf/A is initiating minor compaction (all files) 2023-04-08 20:55:54,767 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 06f3cb8ce0404e826c51a372e2c9bfbf/A in TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:54,767 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/ad3bd7f330584292bbaf3d629f0ba73c, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/9495de42f4874f9582274e9540fd3283, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/2344f51dbb5a4b638bc3098e191b9909, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/a761fe9ad79a430f837a670750aa2bad, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/6bbba4261e0c425392e0c0bc40c4c1f1, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/7dbb9acbed244341ab2bb8faaa27ef77, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/100bd58e18394ee2ae96f06fb8026f62] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp, totalSize=170.1 K 2023-04-08 20:55:54,767 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting ad3bd7f330584292bbaf3d629f0ba73c, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1680987347558 2023-04-08 20:55:54,768 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 81814 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-08 20:55:54,768 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 06f3cb8ce0404e826c51a372e2c9bfbf/B is initiating minor compaction (all files) 2023-04-08 20:55:54,768 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 06f3cb8ce0404e826c51a372e2c9bfbf/B in TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:54,768 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/413c7d407f534a6babd50042971f7427, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/ae54f2a3192f490db7cb70258c4cda70, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/847561cdaf9142faadaa3fa9708abf37, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/4cc3f4af84614958b75109bc42c42070, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/7c3b41df8847405ba6bfad50853493c9, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/adb92f0f261044799a9cb438236b1006, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/2c6107af49e84ec0a53b302bc308eaaa] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp, totalSize=79.9 K 2023-04-08 20:55:54,768 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 413c7d407f534a6babd50042971f7427, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1680987347558 2023-04-08 20:55:54,768 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 9495de42f4874f9582274e9540fd3283, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1680987347880 2023-04-08 20:55:54,769 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting ae54f2a3192f490db7cb70258c4cda70, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1680987347880 2023-04-08 20:55:54,769 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 2344f51dbb5a4b638bc3098e191b9909, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1680987348517 2023-04-08 20:55:54,769 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 847561cdaf9142faadaa3fa9708abf37, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1680987348517 2023-04-08 20:55:54,769 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting a761fe9ad79a430f837a670750aa2bad, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1680987349612 2023-04-08 20:55:54,770 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 4cc3f4af84614958b75109bc42c42070, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1680987349612 2023-04-08 20:55:54,770 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 7c3b41df8847405ba6bfad50853493c9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1680987351766 2023-04-08 20:55:54,770 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 6bbba4261e0c425392e0c0bc40c4c1f1, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1680987351766 2023-04-08 20:55:54,771 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting adb92f0f261044799a9cb438236b1006, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1680987352417 2023-04-08 20:55:54,771 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 7dbb9acbed244341ab2bb8faaa27ef77, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1680987352417 2023-04-08 20:55:54,771 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 2c6107af49e84ec0a53b302bc308eaaa, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680987353571 2023-04-08 20:55:54,773 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 100bd58e18394ee2ae96f06fb8026f62, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680987353571 2023-04-08 20:55:54,789 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 06f3cb8ce0404e826c51a372e2c9bfbf#B#compaction#165 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:54,795 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:54,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:54,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:54,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:54,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:54,831 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:54,831 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:54,831 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:54,831 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:54,832 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,832 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:54,832 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:54,832 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:54,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:54,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:54,833 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:54,833 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:54,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:54,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:54,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:54,834 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:54,851 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:54,851 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:54,851 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:54,851 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:54,851 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:54,857 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/530573f8c3424874b0be4107dfaa1026 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/530573f8c3424874b0be4107dfaa1026 2023-04-08 20:55:54,858 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:54,858 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:54,858 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:54,858 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:55:54,859 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 06f3cb8ce0404e826c51a372e2c9bfbf 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-08 20:55:54,859 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=A 2023-04-08 20:55:54,859 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:54,859 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=B 2023-04-08 20:55:54,859 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:54,860 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=C 2023-04-08 20:55:54,860 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:54,862 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 06f3cb8ce0404e826c51a372e2c9bfbf#A#compaction#166 average throughput is 0.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:54,873 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 06f3cb8ce0404e826c51a372e2c9bfbf/B of 06f3cb8ce0404e826c51a372e2c9bfbf into 530573f8c3424874b0be4107dfaa1026(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:54,873 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:55:54,873 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf., storeName=06f3cb8ce0404e826c51a372e2c9bfbf/B, priority=9, startTime=1680987354762; duration=0sec 2023-04-08 20:55:54,873 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:54,874 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-08 20:55:54,880 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 81814 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-08 20:55:54,880 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 06f3cb8ce0404e826c51a372e2c9bfbf/C is initiating minor compaction (all files) 2023-04-08 20:55:54,880 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 06f3cb8ce0404e826c51a372e2c9bfbf/C in TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:54,880 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/b2f920cf84f24135a0eec58cc131ff29, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/fe2b9554e18b4d6ca180c3af45a12a91, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/d411662c97f140608a40c07b815145b7, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/9248ccf6efa242689421db2bd960dd9a, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/2c529efc887447a28c97802955d6c4ea, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/4cab6128916f459c891cd3ccea997e03, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/988044634b4b4043860fd656ead62259] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp, totalSize=79.9 K 2023-04-08 20:55:54,881 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting b2f920cf84f24135a0eec58cc131ff29, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1680987347558 2023-04-08 20:55:54,886 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting fe2b9554e18b4d6ca180c3af45a12a91, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1680987347880 2023-04-08 20:55:54,887 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting d411662c97f140608a40c07b815145b7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1680987348517 2023-04-08 20:55:54,889 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 9248ccf6efa242689421db2bd960dd9a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1680987349612 2023-04-08 20:55:54,892 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 2c529efc887447a28c97802955d6c4ea, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1680987351766 2023-04-08 20:55:54,892 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 4cab6128916f459c891cd3ccea997e03, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1680987352417 2023-04-08 20:55:54,893 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 988044634b4b4043860fd656ead62259, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680987353571 2023-04-08 20:55:54,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:55:54,901 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:55:54,933 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 06f3cb8ce0404e826c51a372e2c9bfbf#C#compaction#168 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:54,942 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408148ae4b529f4448096016315acbd5180 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408148ae4b529f4448096016315acbd5180 2023-04-08 20:55:54,957 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:54,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41694 deadline: 1680987414953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:55,062 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:55,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41694 deadline: 1680987415059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:55,264 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:55,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41694 deadline: 1680987415263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:55,360 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/84512de0fcc9403c8d3e92a23c941080 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/84512de0fcc9403c8d3e92a23c941080 2023-04-08 20:55:55,364 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=218, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/28fa970266d447469c0993fc2bfc39c5 2023-04-08 20:55:55,366 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 06f3cb8ce0404e826c51a372e2c9bfbf/C of 06f3cb8ce0404e826c51a372e2c9bfbf into 84512de0fcc9403c8d3e92a23c941080(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:55,366 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:55:55,366 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf., storeName=06f3cb8ce0404e826c51a372e2c9bfbf/C, priority=9, startTime=1680987354763; duration=0sec 2023-04-08 20:55:55,367 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:55,566 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:55,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41694 deadline: 1680987415565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:55,765 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/578832365cb54473a44b90ef332213dd as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/578832365cb54473a44b90ef332213dd 2023-04-08 20:55:55,769 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 06f3cb8ce0404e826c51a372e2c9bfbf/A of 06f3cb8ce0404e826c51a372e2c9bfbf into 578832365cb54473a44b90ef332213dd(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-08 20:55:55,769 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:55:55,769 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf., storeName=06f3cb8ce0404e826c51a372e2c9bfbf/A, priority=9, startTime=1680987354762; duration=1sec 2023-04-08 20:55:55,769 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:55,776 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/dff302b9688d46018c7274056fa7b3fc 2023-04-08 20:55:56,071 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:56,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41694 deadline: 1680987416070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:56,185 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/a4f0e0e07e014f029c0f9b4f6b5174da 2023-04-08 20:55:56,190 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/28fa970266d447469c0993fc2bfc39c5 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/28fa970266d447469c0993fc2bfc39c5 2023-04-08 20:55:56,195 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/28fa970266d447469c0993fc2bfc39c5, entries=150, sequenceid=218, filesize=25.3 K 2023-04-08 20:55:56,195 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/dff302b9688d46018c7274056fa7b3fc as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/dff302b9688d46018c7274056fa7b3fc 2023-04-08 20:55:56,200 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/dff302b9688d46018c7274056fa7b3fc, entries=150, sequenceid=218, filesize=11.8 K 2023-04-08 20:55:56,201 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/a4f0e0e07e014f029c0f9b4f6b5174da as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/a4f0e0e07e014f029c0f9b4f6b5174da 2023-04-08 20:55:56,205 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/a4f0e0e07e014f029c0f9b4f6b5174da, entries=150, sequenceid=218, filesize=11.8 K 2023-04-08 20:55:56,206 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 06f3cb8ce0404e826c51a372e2c9bfbf in 1347ms, sequenceid=218, compaction requested=false 2023-04-08 20:55:56,206 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:55:56,206 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:56,207 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:56,207 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:56,207 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:56,207 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:56,207 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:56,208 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:56,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:56,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:56,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:56,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:56,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:56,208 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:56,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:56,209 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:56,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:56,209 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:56,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:56,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:56,209 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:56,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a1309b8[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:56,209 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:56,210 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:56,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:56,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:56,210 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:56,210 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:56,210 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:56,210 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:56,210 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:56,211 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:56,211 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:56,211 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:56,211 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:56,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:56,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:56,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:56,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:56,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:56,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:56,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:56,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:56,214 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:56,214 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:56,214 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:56,214 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:56,214 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:56,215 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:56,216 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:56,217 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:56,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:56,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:56,218 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:56,218 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:56,222 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:56,222 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:56,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:56,222 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:56,222 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:56,222 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:56,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:56,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:56,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:56,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:56,222 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:56,222 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:56,222 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:56,222 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:56,222 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:56,222 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:56,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:56,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:56,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:56,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:56,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:56,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:56,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:56,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:56,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:56,318 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:56,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:56,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:56,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:56,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:56,321 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:56,321 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:56,321 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:56,321 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:56,321 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:56,321 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:56,322 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:56,322 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:56,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:56,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:56,322 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:56,322 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:56,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:56,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:56,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:56,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:56,323 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:56,323 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:56,323 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:56,323 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:56,323 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:56,323 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:56,323 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:56,323 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:56,323 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:56,323 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:55:56,324 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 06f3cb8ce0404e826c51a372e2c9bfbf 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-08 20:55:56,324 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=A 2023-04-08 20:55:56,324 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:56,324 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=B 2023-04-08 20:55:56,324 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:56,324 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=C 2023-04-08 20:55:56,324 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:56,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:55:56,656 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:55:56,680 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:56,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41716 deadline: 1680987416678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:56,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:56,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41660 deadline: 1680987416679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:56,690 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:56,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41702 deadline: 1680987416690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:56,692 DEBUG [Thread-970] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8173 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf., hostname=jenkins-hbase20.apache.org,41091,1680987313601, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 20:55:56,698 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:56,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41688 deadline: 1680987416698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:56,700 DEBUG [Thread-972] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8179 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf., hostname=jenkins-hbase20.apache.org,41091,1680987313601, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 20:55:56,765 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040869bf602fd5484309b01e88982a9d8e00 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040869bf602fd5484309b01e88982a9d8e00 2023-04-08 20:55:56,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:56,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41716 deadline: 1680987416781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:56,785 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:56,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41660 deadline: 1680987416782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:56,986 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:56,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41716 deadline: 1680987416986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:56,986 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:56,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41660 deadline: 1680987416986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,077 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:57,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41694 deadline: 1680987417076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,171 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=238, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/440a642599474ce58f08fccddfee3853 2023-04-08 20:55:57,183 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/3177cea219784386899f0e090c51811f 2023-04-08 20:55:57,194 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/8da4688e0a8a431c923e530cc6fdfb99 2023-04-08 20:55:57,198 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/440a642599474ce58f08fccddfee3853 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/440a642599474ce58f08fccddfee3853 2023-04-08 20:55:57,202 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/440a642599474ce58f08fccddfee3853, entries=150, sequenceid=238, filesize=25.3 K 2023-04-08 20:55:57,202 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/3177cea219784386899f0e090c51811f as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/3177cea219784386899f0e090c51811f 2023-04-08 20:55:57,207 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/3177cea219784386899f0e090c51811f, entries=150, sequenceid=238, filesize=11.8 K 2023-04-08 20:55:57,208 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/8da4688e0a8a431c923e530cc6fdfb99 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/8da4688e0a8a431c923e530cc6fdfb99 2023-04-08 20:55:57,212 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/8da4688e0a8a431c923e530cc6fdfb99, entries=150, sequenceid=238, filesize=11.8 K 2023-04-08 20:55:57,213 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 06f3cb8ce0404e826c51a372e2c9bfbf in 889ms, sequenceid=238, compaction requested=true 2023-04-08 20:55:57,213 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:55:57,213 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:57,213 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:57,213 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:57,213 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,213 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:57,213 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:57,214 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:57,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:57,214 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:57,214 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,214 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:57,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:57,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:57,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:57,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:57,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e1b3cf0[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:57,215 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:57,215 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,216 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,216 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:57,216 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,216 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:57,216 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:57,216 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:57,217 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:57,217 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,217 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:57,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:57,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:57,217 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:57,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:57,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:57,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:57,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:57,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:57,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:57,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:57,219 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:57,219 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:57,219 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:57,219 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:57,219 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:57,220 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,220 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,220 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:57,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:57,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:57,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:57,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:57,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,221 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:57,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:57,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:57,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:57,222 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:57,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:57,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,224 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,224 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:57,224 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:57,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:57,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:57,224 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:57,224 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:57,224 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:57,224 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,224 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,224 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,224 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:57,224 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:57,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:57,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:57,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:57,225 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:57,225 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:57,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:55:57,289 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 06f3cb8ce0404e826c51a372e2c9bfbf 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-08 20:55:57,289 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=A 2023-04-08 20:55:57,289 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:57,289 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=B 2023-04-08 20:55:57,289 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:57,290 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=C 2023-04-08 20:55:57,290 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:57,310 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:57,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41660 deadline: 1680987417308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,312 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:57,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41716 deadline: 1680987417309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,325 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:57,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:57,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:57,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:57,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:57,328 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:57,328 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:57,328 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:57,328 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:57,328 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,328 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:57,329 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,329 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:57,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:57,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:57,329 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,329 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:57,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:57,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:57,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:57,330 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:57,330 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:57,330 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:57,330 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:57,330 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:57,330 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:57,330 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:57,330 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:57,330 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:57,330 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:55:57,331 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:55:57,331 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:57,331 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:57,331 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:57,331 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,331 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:57,331 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:57,331 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,331 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:57,332 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,332 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:57,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:57,333 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:57,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42089747[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:57,333 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,333 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,333 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:57,333 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:57,334 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,334 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:57,334 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:57,334 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,334 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:57,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:57,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:57,334 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:57,334 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:57,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:57,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:57,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:57,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:57,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:57,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:57,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:57,336 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:57,336 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:57,336 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:57,336 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:57,336 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:57,337 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,337 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:57,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:57,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,337 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:57,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:57,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:57,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:57,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:57,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:57,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:57,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:57,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:57,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,343 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,344 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,346 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,347 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:57,347 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:57,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:57,347 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:57,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:57,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:57,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:57,347 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:57,347 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:57,347 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:57,347 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,347 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:57,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:57,347 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:57,347 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,347 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,413 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:57,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41660 deadline: 1680987417411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,414 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:57,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41716 deadline: 1680987417413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,447 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:57,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:57,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:57,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:57,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:57,450 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:57,450 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:57,450 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:57,450 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:57,451 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,451 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:57,451 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,451 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:57,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:57,452 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,452 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:57,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:57,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:57,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:57,452 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:57,452 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:57,452 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:57,452 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:57,453 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:57,453 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:57,453 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:57,453 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:57,453 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:57,453 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:55:57,453 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:55:57,453 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:57,453 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:57,453 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:57,453 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,453 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:57,453 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:57,454 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,454 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:57,454 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,454 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:57,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:57,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69ba7424[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:57,455 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:57,455 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,455 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,456 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:57,456 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,456 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:57,456 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:57,456 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:57,456 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,456 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:57,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:57,456 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:57,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:57,456 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:57,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:57,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:57,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:57,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:57,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:57,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:57,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:57,458 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:57,458 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:57,458 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:57,458 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:57,458 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:57,460 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,461 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:57,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:57,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:57,461 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:57,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:57,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:57,468 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:57,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:57,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:57,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,469 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:57,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:57,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,480 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:57,480 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,481 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:57,481 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:57,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:57,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:57,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:57,481 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:57,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:57,481 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:57,481 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:57,481 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,481 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:57,481 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,481 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,481 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:57,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:57,581 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:57,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:57,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:57,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:57,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:57,584 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:57,584 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:57,584 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:57,584 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:57,584 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,584 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:57,585 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,585 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:57,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:57,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:57,586 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,586 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:57,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:57,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:57,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:57,586 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:57,587 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:57,587 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:57,587 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:57,587 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:57,587 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:57,587 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:57,587 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:57,587 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:57,587 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:55:57,587 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:55:57,587 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:57,587 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:57,587 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:57,587 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,587 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:57,587 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:57,588 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,588 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:57,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:57,588 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:57,588 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:57,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:57,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:57,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:57,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:57,589 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:57,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b06ee62[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:57,590 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,590 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,591 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:57,591 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:57,591 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:57,598 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,598 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:57,602 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,602 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:57,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:57,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:57,602 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:57,602 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:57,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:57,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:57,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:57,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:57,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:57,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:57,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:57,605 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:57,605 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:57,605 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:57,605 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:57,605 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:57,606 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,606 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:57,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:57,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,606 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:57,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:57,607 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:57,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:57,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:57,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:57,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:57,608 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:57,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:57,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,609 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,609 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:57,609 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:57,609 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,609 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:57,609 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:57,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:57,609 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:57,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:57,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:57,609 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:57,610 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:57,610 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:57,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:57,610 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,610 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:57,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:57,616 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:57,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41660 deadline: 1680987417614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,617 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:57,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41716 deadline: 1680987417616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,706 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408725834a0badf4803b03584f4bc4e1622 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408725834a0badf4803b03584f4bc4e1622 2023-04-08 20:55:57,710 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:57,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:57,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:57,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:57,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:57,726 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:57,726 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:57,726 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:57,726 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:57,726 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,726 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:57,727 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:57,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:57,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:57,727 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,728 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=254, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/17ce95c063f84bf4aee8d2d53d245cb2 2023-04-08 20:55:57,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:57,729 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,729 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:57,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:57,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:57,730 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:57,731 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:57,731 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:57,731 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:57,731 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:57,731 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:57,734 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:57,734 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:57,734 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:57,734 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:55:57,734 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:55:57,734 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:57,734 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:57,734 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:57,734 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,742 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:57,742 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:57,744 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,744 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:57,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:57,744 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,744 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:57,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:57,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:57,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:57,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:57,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:57,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c3dbf4e[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:57,745 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:57,745 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,746 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,746 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:57,746 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:57,746 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,746 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:57,746 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:57,747 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:57,747 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,747 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:57,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:57,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:57,747 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:57,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:57,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:57,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:57,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:57,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:57,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:57,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:57,749 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:57,749 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:57,749 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:57,749 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:57,749 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:57,750 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,750 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:57,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,750 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:57,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:57,751 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:57,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:57,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:57,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,753 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:57,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:57,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,756 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,756 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:57,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:57,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:57,756 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:57,757 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:57,756 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:57,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:57,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:57,756 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:57,757 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,757 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,757 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:57,757 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:57,757 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:57,757 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:57,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:57,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:57,776 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/354f83ccde8644eeaeda4bf6d9a6698b 2023-04-08 20:55:57,830 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/dfad5af107d24fe995ef37d0e54d53d5 2023-04-08 20:55:57,836 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/17ce95c063f84bf4aee8d2d53d245cb2 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/17ce95c063f84bf4aee8d2d53d245cb2 2023-04-08 20:55:57,841 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/17ce95c063f84bf4aee8d2d53d245cb2, entries=150, sequenceid=254, filesize=25.3 K 2023-04-08 20:55:57,842 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/354f83ccde8644eeaeda4bf6d9a6698b as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/354f83ccde8644eeaeda4bf6d9a6698b 2023-04-08 20:55:57,848 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/354f83ccde8644eeaeda4bf6d9a6698b, entries=150, sequenceid=254, filesize=11.8 K 2023-04-08 20:55:57,849 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/dfad5af107d24fe995ef37d0e54d53d5 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/dfad5af107d24fe995ef37d0e54d53d5 2023-04-08 20:55:57,855 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/dfad5af107d24fe995ef37d0e54d53d5, entries=150, sequenceid=254, filesize=11.8 K 2023-04-08 20:55:57,857 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:57,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:57,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:57,864 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 06f3cb8ce0404e826c51a372e2c9bfbf in 575ms, sequenceid=254, compaction requested=true 2023-04-08 20:55:57,864 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:55:57,864 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:57,864 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 20:55:57,864 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 20:55:57,864 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:57,864 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:57,865 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104028 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 20:55:57,865 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48588 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 20:55:57,866 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): 06f3cb8ce0404e826c51a372e2c9bfbf/A is initiating minor compaction (all files) 2023-04-08 20:55:57,866 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 06f3cb8ce0404e826c51a372e2c9bfbf/B is initiating minor compaction (all files) 2023-04-08 20:55:57,866 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 06f3cb8ce0404e826c51a372e2c9bfbf/A in TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:57,866 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 06f3cb8ce0404e826c51a372e2c9bfbf/B in TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:57,866 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/578832365cb54473a44b90ef332213dd, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/28fa970266d447469c0993fc2bfc39c5, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/440a642599474ce58f08fccddfee3853, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/17ce95c063f84bf4aee8d2d53d245cb2] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp, totalSize=101.6 K 2023-04-08 20:55:57,866 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/530573f8c3424874b0be4107dfaa1026, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/dff302b9688d46018c7274056fa7b3fc, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/3177cea219784386899f0e090c51811f, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/354f83ccde8644eeaeda4bf6d9a6698b] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp, totalSize=47.4 K 2023-04-08 20:55:57,866 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 530573f8c3424874b0be4107dfaa1026, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680987353571 2023-04-08 20:55:57,866 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 578832365cb54473a44b90ef332213dd, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680987353571 2023-04-08 20:55:57,867 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting dff302b9688d46018c7274056fa7b3fc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680987354274 2023-04-08 20:55:57,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:57,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:57,867 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 28fa970266d447469c0993fc2bfc39c5, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680987354274 2023-04-08 20:55:57,867 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 3177cea219784386899f0e090c51811f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1680987354923 2023-04-08 20:55:57,867 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:57,867 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:57,867 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:57,867 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:57,867 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 440a642599474ce58f08fccddfee3853, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1680987354923 2023-04-08 20:55:57,868 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,868 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:57,868 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 354f83ccde8644eeaeda4bf6d9a6698b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1680987356676 2023-04-08 20:55:57,868 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 17ce95c063f84bf4aee8d2d53d245cb2, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1680987356676 2023-04-08 20:55:57,868 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:57,868 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:57,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:57,869 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,869 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:57,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:57,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:57,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:57,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:57,869 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:57,869 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:57,869 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:57,870 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:57,870 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:57,870 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:57,870 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:57,870 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:57,870 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:57,870 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:55:57,870 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 06f3cb8ce0404e826c51a372e2c9bfbf 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-08 20:55:57,870 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=A 2023-04-08 20:55:57,870 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:57,870 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=B 2023-04-08 20:55:57,870 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:57,870 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=C 2023-04-08 20:55:57,871 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:57,879 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 06f3cb8ce0404e826c51a372e2c9bfbf#B#compaction#177 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:57,896 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 06f3cb8ce0404e826c51a372e2c9bfbf#A#compaction#178 average throughput is 1.10 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:57,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:55:57,919 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:55:57,940 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:57,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41716 deadline: 1680987417937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:57,941 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:57,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41660 deadline: 1680987417938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:58,044 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:58,044 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:58,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41716 deadline: 1680987418042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:58,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41660 deadline: 1680987418042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:58,248 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:58,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41660 deadline: 1680987418246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:58,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:58,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41716 deadline: 1680987418246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:58,298 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408bc54aa4804974b3abff8daf34ce62924 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408bc54aa4804974b3abff8daf34ce62924 2023-04-08 20:55:58,299 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/5abb39f4e41d48f984183f3a91cfa414 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/5abb39f4e41d48f984183f3a91cfa414 2023-04-08 20:55:58,304 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 06f3cb8ce0404e826c51a372e2c9bfbf/B of 06f3cb8ce0404e826c51a372e2c9bfbf into 5abb39f4e41d48f984183f3a91cfa414(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:58,304 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:55:58,304 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf., storeName=06f3cb8ce0404e826c51a372e2c9bfbf/B, priority=12, startTime=1680987357864; duration=0sec 2023-04-08 20:55:58,304 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:55:58,304 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 20:55:58,306 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48588 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 20:55:58,306 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 06f3cb8ce0404e826c51a372e2c9bfbf/C is initiating minor compaction (all files) 2023-04-08 20:55:58,306 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 06f3cb8ce0404e826c51a372e2c9bfbf/C in TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:58,306 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/84512de0fcc9403c8d3e92a23c941080, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/a4f0e0e07e014f029c0f9b4f6b5174da, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/8da4688e0a8a431c923e530cc6fdfb99, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/dfad5af107d24fe995ef37d0e54d53d5] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp, totalSize=47.4 K 2023-04-08 20:55:58,308 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 84512de0fcc9403c8d3e92a23c941080, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680987353571 2023-04-08 20:55:58,309 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting a4f0e0e07e014f029c0f9b4f6b5174da, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680987354274 2023-04-08 20:55:58,310 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 8da4688e0a8a431c923e530cc6fdfb99, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1680987354923 2023-04-08 20:55:58,310 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting dfad5af107d24fe995ef37d0e54d53d5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1680987356676 2023-04-08 20:55:58,317 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 06f3cb8ce0404e826c51a372e2c9bfbf#C#compaction#180 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:55:58,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:58,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41660 deadline: 1680987418549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:58,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:58,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41716 deadline: 1680987418549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:58,715 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=270, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/f42c11af18ec4e81934f4f776bd4882f 2023-04-08 20:55:58,730 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/5bb49fe6b3ee46aca0084d2f6b76be45 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/5bb49fe6b3ee46aca0084d2f6b76be45 2023-04-08 20:55:58,731 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=270 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/9631714bc9e54b658a95e0ba074e2c4a 2023-04-08 20:55:58,739 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/d992eb4846bb4f629e9fee232b2e62f3 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/d992eb4846bb4f629e9fee232b2e62f3 2023-04-08 20:55:58,741 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 06f3cb8ce0404e826c51a372e2c9bfbf/C of 06f3cb8ce0404e826c51a372e2c9bfbf into 5bb49fe6b3ee46aca0084d2f6b76be45(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:58,741 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:55:58,741 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf., storeName=06f3cb8ce0404e826c51a372e2c9bfbf/C, priority=12, startTime=1680987357864; duration=0sec 2023-04-08 20:55:58,741 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:58,749 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 06f3cb8ce0404e826c51a372e2c9bfbf/A of 06f3cb8ce0404e826c51a372e2c9bfbf into d992eb4846bb4f629e9fee232b2e62f3(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:55:58,749 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:55:58,750 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf., storeName=06f3cb8ce0404e826c51a372e2c9bfbf/A, priority=12, startTime=1680987357864; duration=0sec 2023-04-08 20:55:58,750 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:55:58,760 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=270 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/727e8e1bd59e4890b28e00388ce84f5f 2023-04-08 20:55:58,766 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/f42c11af18ec4e81934f4f776bd4882f as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/f42c11af18ec4e81934f4f776bd4882f 2023-04-08 20:55:58,772 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/f42c11af18ec4e81934f4f776bd4882f, entries=150, sequenceid=270, filesize=25.4 K 2023-04-08 20:55:58,773 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/9631714bc9e54b658a95e0ba074e2c4a as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/9631714bc9e54b658a95e0ba074e2c4a 2023-04-08 20:55:58,778 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/9631714bc9e54b658a95e0ba074e2c4a, entries=150, sequenceid=270, filesize=11.9 K 2023-04-08 20:55:58,780 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/727e8e1bd59e4890b28e00388ce84f5f as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/727e8e1bd59e4890b28e00388ce84f5f 2023-04-08 20:55:58,791 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/727e8e1bd59e4890b28e00388ce84f5f, entries=150, sequenceid=270, filesize=11.9 K 2023-04-08 20:55:58,792 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 06f3cb8ce0404e826c51a372e2c9bfbf in 922ms, sequenceid=270, compaction requested=false 2023-04-08 20:55:58,792 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:55:58,792 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:58,792 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:58,792 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:58,792 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:58,792 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:58,792 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:58,794 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:58,794 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:58,794 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:58,795 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:58,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:58,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:58,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:58,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:58,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:58,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:58,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:58,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:58,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:58,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a09ab21[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:58,798 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:58,798 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:58,799 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:58,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:58,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:58,799 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:58,799 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:58,799 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:58,799 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:58,800 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:58,801 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:58,801 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:58,801 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:58,801 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:58,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:58,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:58,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:58,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:58,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:58,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:58,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:58,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:58,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:58,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:58,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:58,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:58,804 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:58,804 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:58,804 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:58,804 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:58,804 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:58,806 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:58,806 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:58,806 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:58,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:58,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:58,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:58,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:58,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:58,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:58,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:58,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:58,808 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:58,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:58,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:58,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:58,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:58,809 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:58,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:58,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:58,822 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:58,822 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:58,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:58,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:58,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:58,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:58,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:58,822 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:58,822 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:58,823 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:58,823 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:58,823 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:58,823 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:58,823 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:58,823 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:58,823 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:58,823 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:58,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:58,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:58,923 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:58,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:58,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:58,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:58,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:58,926 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:58,926 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:58,926 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:58,926 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:58,926 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:58,926 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:58,927 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:58,927 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:58,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:58,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:58,927 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:58,927 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:58,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:58,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:58,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:58,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:58,928 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:58,928 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:58,928 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:58,928 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:58,928 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:58,928 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:58,928 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:58,928 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:58,928 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:58,928 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:55:58,928 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 06f3cb8ce0404e826c51a372e2c9bfbf 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-08 20:55:58,929 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=A 2023-04-08 20:55:58,929 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:58,929 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=B 2023-04-08 20:55:58,929 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:58,929 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=C 2023-04-08 20:55:58,929 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:58,944 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304083f82879b45a54ca88282e27508bf1543 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304083f82879b45a54ca88282e27508bf1543 2023-04-08 20:55:59,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:55:59,056 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:55:59,078 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:59,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41660 deadline: 1680987419076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:59,078 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:59,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41716 deadline: 1680987419076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:59,082 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:59,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41694 deadline: 1680987419079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:59,095 DEBUG [Thread-974] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4141 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf., hostname=jenkins-hbase20.apache.org,41091,1680987313601, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 20:55:59,181 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:59,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41660 deadline: 1680987419179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:59,181 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:59,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41716 deadline: 1680987419179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:59,353 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=289, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/22142b4928624077a087e984e03db47c 2023-04-08 20:55:59,383 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:59,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41716 deadline: 1680987419382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:59,384 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:59,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41660 deadline: 1680987419383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:59,686 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:59,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41716 deadline: 1680987419685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:59,687 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:55:59,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41660 deadline: 1680987419686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:59,763 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=289 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/a520694bf2844712b9fb0b4352f3aa0d 2023-04-08 20:55:59,776 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=289 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/83d23b3108f14d23a0a5bd50f88062d8 2023-04-08 20:55:59,786 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/22142b4928624077a087e984e03db47c as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/22142b4928624077a087e984e03db47c 2023-04-08 20:55:59,792 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/22142b4928624077a087e984e03db47c, entries=150, sequenceid=289, filesize=25.4 K 2023-04-08 20:55:59,793 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/a520694bf2844712b9fb0b4352f3aa0d as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/a520694bf2844712b9fb0b4352f3aa0d 2023-04-08 20:55:59,798 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/a520694bf2844712b9fb0b4352f3aa0d, entries=150, sequenceid=289, filesize=11.9 K 2023-04-08 20:55:59,799 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/83d23b3108f14d23a0a5bd50f88062d8 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/83d23b3108f14d23a0a5bd50f88062d8 2023-04-08 20:55:59,803 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/83d23b3108f14d23a0a5bd50f88062d8, entries=150, sequenceid=289, filesize=11.9 K 2023-04-08 20:55:59,803 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 06f3cb8ce0404e826c51a372e2c9bfbf in 875ms, sequenceid=289, compaction requested=true 2023-04-08 20:55:59,804 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:55:59,804 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:59,804 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:55:59,804 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:55:59,804 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:59,808 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:55:59,808 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:55:59,810 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:59,810 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:59,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:59,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:59,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:59,810 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:59,810 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:55:59,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:59,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:59,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:59,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:59,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:59,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:55:59,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70a8fd10[Count = 0] remaining members to acquire global barrier 2023-04-08 20:55:59,812 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:55:59,812 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:59,812 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:59,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:59,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:59,813 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:55:59,813 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:59,813 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:55:59,813 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:55:59,813 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:55:59,814 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:55:59,814 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:59,814 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:55:59,814 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:59,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:59,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:59,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:59,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:59,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:59,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:59,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:59,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:59,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:59,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:59,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:55:59,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:55:59,817 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:55:59,817 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:55:59,817 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:55:59,817 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:55:59,817 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:55:59,818 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:59,818 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:59,818 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:59,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:59,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:59,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:59,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:55:59,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:55:59,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:59,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:59,819 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:59,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:55:59,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:59,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:59,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:55:59,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:59,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:59,820 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:59,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:55:59,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:55:59,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:59,822 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:59,823 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:55:59,823 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:59,823 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:59,823 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:59,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:55:59,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:55:59,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:59,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:59,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:55:59,823 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:59,823 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:59,823 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:55:59,823 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:59,823 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:55:59,823 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:55:59,823 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:55:59,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:55:59,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:55:59,923 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:55:59,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:55:59,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:55:59,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:55:59,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:55:59,926 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:55:59,926 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:59,926 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:55:59,926 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:55:59,926 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:59,926 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:59,927 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:55:59,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:55:59,927 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:59,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:55:59,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:59,927 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:55:59,927 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:55:59,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:55:59,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:55:59,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:55:59,928 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:55:59,929 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:55:59,929 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:55:59,929 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:55:59,929 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:55:59,929 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:55:59,929 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:55:59,929 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:55:59,929 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:55:59,929 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:55:59,929 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 06f3cb8ce0404e826c51a372e2c9bfbf 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-08 20:55:59,929 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=A 2023-04-08 20:55:59,930 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:59,930 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=B 2023-04-08 20:55:59,930 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:55:59,930 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=C 2023-04-08 20:55:59,930 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:00,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:56:00,194 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:56:00,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:00,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41660 deadline: 1680987420230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:00,234 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:00,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41716 deadline: 1680987420230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:00,336 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:00,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41660 deadline: 1680987420335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:00,336 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:00,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41716 deadline: 1680987420335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:00,352 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040872a055dd7faf4a56aface8c527242cb0 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040872a055dd7faf4a56aface8c527242cb0 2023-04-08 20:56:00,540 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:00,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41716 deadline: 1680987420537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:00,540 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:00,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41660 deadline: 1680987420538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:00,768 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=305, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/104af472bfa745858ad76c82328bc062 2023-04-08 20:56:00,842 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:00,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41716 deadline: 1680987420841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:00,842 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:00,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41660 deadline: 1680987420841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:01,192 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=305 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/80b90c0344594afa9c09f37bd0983fcb 2023-04-08 20:56:01,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:01,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41716 deadline: 1680987421345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:01,347 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:01,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41660 deadline: 1680987421346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:01,601 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=305 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/db170529132a4c46828f0a957edfacf6 2023-04-08 20:56:01,606 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/104af472bfa745858ad76c82328bc062 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/104af472bfa745858ad76c82328bc062 2023-04-08 20:56:01,611 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/104af472bfa745858ad76c82328bc062, entries=150, sequenceid=305, filesize=25.4 K 2023-04-08 20:56:01,612 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/80b90c0344594afa9c09f37bd0983fcb as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/80b90c0344594afa9c09f37bd0983fcb 2023-04-08 20:56:01,618 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/80b90c0344594afa9c09f37bd0983fcb, entries=150, sequenceid=305, filesize=11.9 K 2023-04-08 20:56:01,619 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/db170529132a4c46828f0a957edfacf6 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/db170529132a4c46828f0a957edfacf6 2023-04-08 20:56:01,630 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/db170529132a4c46828f0a957edfacf6, entries=150, sequenceid=305, filesize=11.9 K 2023-04-08 20:56:01,631 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 06f3cb8ce0404e826c51a372e2c9bfbf in 1702ms, sequenceid=305, compaction requested=true 2023-04-08 20:56:01,631 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:56:01,631 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:56:01,631 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:01,631 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:01,631 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:01,631 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:01,631 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:01,632 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:01,632 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:01,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:01,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:01,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:01,633 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:01,633 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:01,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:01,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:01,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:01,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:01,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:01,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:01,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c617899[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:01,643 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:01,643 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:01,647 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:01,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:01,647 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:01,647 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:01,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:01,647 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:01,647 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:01,647 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:01,648 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:01,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:01,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:01,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:01,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:01,649 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:01,649 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:01,649 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:01,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:01,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:01,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:01,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:01,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:01,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:01,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:01,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:01,651 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:01,651 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:01,651 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:01,651 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:01,651 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:01,652 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:01,652 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:01,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:01,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:01,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:01,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:01,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:01,652 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:01,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:01,653 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:01,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:01,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:01,654 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:01,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:01,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:01,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:01,655 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:01,655 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:01,655 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:01,656 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:01,656 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:01,656 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:01,656 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:01,656 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:01,656 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:01,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:01,656 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:01,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:01,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:01,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:01,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:01,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:01,658 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:01,658 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:01,758 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:01,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:01,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:01,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:01,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:01,766 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:01,766 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:01,766 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:01,766 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:01,766 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:01,766 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:01,768 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:01,768 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:01,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:01,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:01,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:01,769 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:01,769 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:01,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:01,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:01,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:01,770 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:01,770 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:01,770 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:01,770 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:01,770 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:01,770 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:01,774 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:01,774 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:01,774 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:56:01,774 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:56:01,774 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 06f3cb8ce0404e826c51a372e2c9bfbf 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-08 20:56:01,774 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=A 2023-04-08 20:56:01,775 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:01,775 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=B 2023-04-08 20:56:01,775 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:01,775 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=C 2023-04-08 20:56:01,775 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:02,211 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408cdb2a11277b5471bbe38c3e98c783f8e to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408cdb2a11277b5471bbe38c3e98c783f8e 2023-04-08 20:56:02,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:56:02,358 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:56:02,385 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:02,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41716 deadline: 1680987422380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:02,385 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:02,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41660 deadline: 1680987422381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:02,487 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:02,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41660 deadline: 1680987422487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:02,494 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:02,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41716 deadline: 1680987422493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:02,627 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=322, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/c3c91574c83a40c5933063cabf762db2 2023-04-08 20:56:02,692 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:02,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41660 deadline: 1680987422690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:02,705 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:02,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41716 deadline: 1680987422698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:02,997 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:02,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41660 deadline: 1680987422994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,014 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:03,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41716 deadline: 1680987423009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,052 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/d02f19c1dd2d431cbc78f26be161a464 2023-04-08 20:56:03,097 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:03,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41694 deadline: 1680987423096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,099 DEBUG [Thread-974] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8144 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf., hostname=jenkins-hbase20.apache.org,41091,1680987313601, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 20:56:03,461 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/406ebedd98d348ef956f530ba005846c 2023-04-08 20:56:03,466 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/c3c91574c83a40c5933063cabf762db2 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/c3c91574c83a40c5933063cabf762db2 2023-04-08 20:56:03,470 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/c3c91574c83a40c5933063cabf762db2, entries=150, sequenceid=322, filesize=25.4 K 2023-04-08 20:56:03,471 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/d02f19c1dd2d431cbc78f26be161a464 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/d02f19c1dd2d431cbc78f26be161a464 2023-04-08 20:56:03,479 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/d02f19c1dd2d431cbc78f26be161a464, entries=150, sequenceid=322, filesize=11.9 K 2023-04-08 20:56:03,480 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/406ebedd98d348ef956f530ba005846c as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/406ebedd98d348ef956f530ba005846c 2023-04-08 20:56:03,484 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/406ebedd98d348ef956f530ba005846c, entries=150, sequenceid=322, filesize=11.9 K 2023-04-08 20:56:03,484 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 06f3cb8ce0404e826c51a372e2c9bfbf in 1710ms, sequenceid=322, compaction requested=true 2023-04-08 20:56:03,485 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:56:03,485 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:56:03,485 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:03,485 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:03,485 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,485 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:03,485 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:03,487 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,487 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:03,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:03,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:03,488 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:03,488 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:03,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:03,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:03,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:03,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:03,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:03,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d076994[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:03,489 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:03,489 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:03,489 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:03,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:03,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:03,489 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,489 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:03,489 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:03,490 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:03,490 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:03,490 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,490 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:03,490 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:03,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:03,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:03,490 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:03,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:03,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:03,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:03,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:03,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:03,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:03,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:03,493 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:03,493 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:03,493 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:03,493 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:03,493 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:03,493 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:03,494 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:03,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:03,494 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:03,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:03,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:03,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:03,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:03,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:03,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:03,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:03,494 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:03,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:03,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:03,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:03,495 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:03,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:03,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,497 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:03,497 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:03,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:03,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:03,497 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,497 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:03,497 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:03,497 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:03,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:03,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:03,497 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,497 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:03,497 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:03,497 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:03,497 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:03,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:03,497 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:03,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:03,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:03,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:56:03,502 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 06f3cb8ce0404e826c51a372e2c9bfbf 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-08 20:56:03,502 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=A 2023-04-08 20:56:03,518 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:03,519 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=B 2023-04-08 20:56:03,519 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:03,519 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=C 2023-04-08 20:56:03,519 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:03,538 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040896a263d4a02b44158aefe2c0ab6fe65e to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040896a263d4a02b44158aefe2c0ab6fe65e 2023-04-08 20:56:03,543 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=338, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/e20fac8da609434d8341cf0084a04c71 2023-04-08 20:56:03,550 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:03,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41660 deadline: 1680987423545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:03,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41716 deadline: 1680987423548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,555 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/e0e6c4c7b8f3410b93eb87b9be62dd66 2023-04-08 20:56:03,566 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/52698dad24f14bf6be66fc201fabc0c4 2023-04-08 20:56:03,571 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/e20fac8da609434d8341cf0084a04c71 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/e20fac8da609434d8341cf0084a04c71 2023-04-08 20:56:03,575 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/e20fac8da609434d8341cf0084a04c71, entries=200, sequenceid=338, filesize=32.3 K 2023-04-08 20:56:03,575 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/e0e6c4c7b8f3410b93eb87b9be62dd66 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/e0e6c4c7b8f3410b93eb87b9be62dd66 2023-04-08 20:56:03,579 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/e0e6c4c7b8f3410b93eb87b9be62dd66, entries=150, sequenceid=338, filesize=11.9 K 2023-04-08 20:56:03,583 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/52698dad24f14bf6be66fc201fabc0c4 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/52698dad24f14bf6be66fc201fabc0c4 2023-04-08 20:56:03,588 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/52698dad24f14bf6be66fc201fabc0c4, entries=150, sequenceid=338, filesize=11.9 K 2023-04-08 20:56:03,589 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 06f3cb8ce0404e826c51a372e2c9bfbf in 87ms, sequenceid=338, compaction requested=true 2023-04-08 20:56:03,589 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:56:03,589 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:56:03,589 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-08 20:56:03,589 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-08 20:56:03,589 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:56:03,590 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:56:03,593 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 163746 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-08 20:56:03,593 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73546 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-08 20:56:03,593 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): 06f3cb8ce0404e826c51a372e2c9bfbf/A is initiating minor compaction (all files) 2023-04-08 20:56:03,593 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 06f3cb8ce0404e826c51a372e2c9bfbf/B is initiating minor compaction (all files) 2023-04-08 20:56:03,593 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 06f3cb8ce0404e826c51a372e2c9bfbf/A in TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:56:03,593 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 06f3cb8ce0404e826c51a372e2c9bfbf/B in TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:56:03,593 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/5abb39f4e41d48f984183f3a91cfa414, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/9631714bc9e54b658a95e0ba074e2c4a, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/a520694bf2844712b9fb0b4352f3aa0d, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/80b90c0344594afa9c09f37bd0983fcb, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/d02f19c1dd2d431cbc78f26be161a464, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/e0e6c4c7b8f3410b93eb87b9be62dd66] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp, totalSize=71.8 K 2023-04-08 20:56:03,593 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/d992eb4846bb4f629e9fee232b2e62f3, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/f42c11af18ec4e81934f4f776bd4882f, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/22142b4928624077a087e984e03db47c, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/104af472bfa745858ad76c82328bc062, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/c3c91574c83a40c5933063cabf762db2, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/e20fac8da609434d8341cf0084a04c71] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp, totalSize=159.9 K 2023-04-08 20:56:03,594 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 5abb39f4e41d48f984183f3a91cfa414, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1680987356676 2023-04-08 20:56:03,594 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting d992eb4846bb4f629e9fee232b2e62f3, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1680987356676 2023-04-08 20:56:03,594 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 9631714bc9e54b658a95e0ba074e2c4a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1680987357305 2023-04-08 20:56:03,595 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting f42c11af18ec4e81934f4f776bd4882f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1680987357305 2023-04-08 20:56:03,595 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting a520694bf2844712b9fb0b4352f3aa0d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1680987357932 2023-04-08 20:56:03,595 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 22142b4928624077a087e984e03db47c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1680987357932 2023-04-08 20:56:03,595 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 80b90c0344594afa9c09f37bd0983fcb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1680987359063 2023-04-08 20:56:03,595 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 104af472bfa745858ad76c82328bc062, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1680987359063 2023-04-08 20:56:03,596 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting d02f19c1dd2d431cbc78f26be161a464, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1680987360210 2023-04-08 20:56:03,596 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting c3c91574c83a40c5933063cabf762db2, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1680987360210 2023-04-08 20:56:03,596 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting e0e6c4c7b8f3410b93eb87b9be62dd66, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1680987362373 2023-04-08 20:56:03,596 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting e20fac8da609434d8341cf0084a04c71, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1680987362373 2023-04-08 20:56:03,598 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:03,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:03,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:03,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:03,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:03,602 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:03,602 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:03,602 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:03,602 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:03,602 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:03,602 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:03,603 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:03,603 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:03,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:03,604 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,604 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:03,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:03,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:03,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:03,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:03,605 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:03,605 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:03,605 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:03,605 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:03,606 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:03,614 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:03,614 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:03,614 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:56:03,614 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:56:03,614 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 06f3cb8ce0404e826c51a372e2c9bfbf 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-08 20:56:03,615 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=A 2023-04-08 20:56:03,615 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:03,615 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=B 2023-04-08 20:56:03,615 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:03,615 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=C 2023-04-08 20:56:03,615 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:03,619 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 06f3cb8ce0404e826c51a372e2c9bfbf#B#compaction#195 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:56:03,636 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 06f3cb8ce0404e826c51a372e2c9bfbf#A#compaction#196 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:56:03,647 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/d46cf04703a3412db7f81592ed55faa1 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/d46cf04703a3412db7f81592ed55faa1 2023-04-08 20:56:03,654 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 06f3cb8ce0404e826c51a372e2c9bfbf/B of 06f3cb8ce0404e826c51a372e2c9bfbf into d46cf04703a3412db7f81592ed55faa1(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:56:03,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:56:03,654 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:56:03,654 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:56:03,654 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf., storeName=06f3cb8ce0404e826c51a372e2c9bfbf/B, priority=10, startTime=1680987363589; duration=0sec 2023-04-08 20:56:03,655 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:56:03,655 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-08 20:56:03,656 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73546 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-08 20:56:03,656 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 06f3cb8ce0404e826c51a372e2c9bfbf/C is initiating minor compaction (all files) 2023-04-08 20:56:03,657 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 06f3cb8ce0404e826c51a372e2c9bfbf/C in TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:56:03,657 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/5bb49fe6b3ee46aca0084d2f6b76be45, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/727e8e1bd59e4890b28e00388ce84f5f, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/83d23b3108f14d23a0a5bd50f88062d8, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/db170529132a4c46828f0a957edfacf6, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/406ebedd98d348ef956f530ba005846c, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/52698dad24f14bf6be66fc201fabc0c4] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp, totalSize=71.8 K 2023-04-08 20:56:03,657 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 5bb49fe6b3ee46aca0084d2f6b76be45, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1680987356676 2023-04-08 20:56:03,658 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 727e8e1bd59e4890b28e00388ce84f5f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1680987357305 2023-04-08 20:56:03,658 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 83d23b3108f14d23a0a5bd50f88062d8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1680987357932 2023-04-08 20:56:03,665 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting db170529132a4c46828f0a957edfacf6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1680987359063 2023-04-08 20:56:03,668 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 406ebedd98d348ef956f530ba005846c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1680987360210 2023-04-08 20:56:03,674 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 52698dad24f14bf6be66fc201fabc0c4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1680987362373 2023-04-08 20:56:03,682 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408a826f5e2e4b04cc8ad3d620aa22d691b to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408a826f5e2e4b04cc8ad3d620aa22d691b 2023-04-08 20:56:03,683 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:03,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41660 deadline: 1680987423679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,687 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:03,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41716 deadline: 1680987423681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,696 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 06f3cb8ce0404e826c51a372e2c9bfbf#C#compaction#198 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:56:03,714 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=354, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/63367cea9b4d463f845f664ab323ddcb 2023-04-08 20:56:03,719 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/63da166db7f84fd788ca9c4760bba13a as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/63da166db7f84fd788ca9c4760bba13a 2023-04-08 20:56:03,732 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 06f3cb8ce0404e826c51a372e2c9bfbf/C of 06f3cb8ce0404e826c51a372e2c9bfbf into 63da166db7f84fd788ca9c4760bba13a(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:56:03,732 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:56:03,733 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf., storeName=06f3cb8ce0404e826c51a372e2c9bfbf/C, priority=10, startTime=1680987363590; duration=0sec 2023-04-08 20:56:03,733 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:56:03,745 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=354 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/cb16e8a3fe9542b08c34270ba5cb4f45 2023-04-08 20:56:03,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:03,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41660 deadline: 1680987423785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,790 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:03,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41716 deadline: 1680987423788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,808 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=354 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/1778cacb9c70493897fd84b968d5427e 2023-04-08 20:56:03,815 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/63367cea9b4d463f845f664ab323ddcb as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/63367cea9b4d463f845f664ab323ddcb 2023-04-08 20:56:03,820 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/63367cea9b4d463f845f664ab323ddcb, entries=150, sequenceid=354, filesize=25.4 K 2023-04-08 20:56:03,822 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/cb16e8a3fe9542b08c34270ba5cb4f45 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/cb16e8a3fe9542b08c34270ba5cb4f45 2023-04-08 20:56:03,831 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/cb16e8a3fe9542b08c34270ba5cb4f45, entries=150, sequenceid=354, filesize=11.9 K 2023-04-08 20:56:03,833 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/1778cacb9c70493897fd84b968d5427e as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/1778cacb9c70493897fd84b968d5427e 2023-04-08 20:56:03,838 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/1778cacb9c70493897fd84b968d5427e, entries=150, sequenceid=354, filesize=11.9 K 2023-04-08 20:56:03,839 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 06f3cb8ce0404e826c51a372e2c9bfbf in 225ms, sequenceid=354, compaction requested=false 2023-04-08 20:56:03,839 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:56:03,839 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:56:03,839 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:03,839 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:03,839 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,842 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:03,842 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:03,843 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,843 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:03,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:03,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:03,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:03,844 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:03,844 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:03,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:03,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:03,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:03,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:03,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ad097b6[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:03,845 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:03,845 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:03,846 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:03,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:03,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:03,846 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:03,846 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:03,846 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:03,846 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,846 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:03,847 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:03,847 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:03,847 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,847 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:03,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:03,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:03,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:03,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:03,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:03,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:03,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:03,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:03,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:03,850 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:03,850 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:03,850 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:03,850 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:03,850 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:03,851 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:03,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:03,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:03,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:03,852 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:03,852 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:03,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:03,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:03,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:03,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:03,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:03,852 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:03,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:03,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:03,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:03,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:03,854 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:03,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,856 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:03,856 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:03,856 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:03,857 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:03,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:03,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:03,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:03,857 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:03,857 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:03,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:03,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:03,857 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:03,858 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:03,858 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,858 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:03,858 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:03,858 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:03,957 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:03,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:03,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:03,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:03,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:03,961 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:03,961 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:03,961 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:03,962 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:03,963 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:03,963 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:03,964 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:03,964 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:03,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:03,964 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:03,964 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:03,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:03,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:03,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:03,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:03,968 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:03,970 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:03,970 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:03,970 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:03,970 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:03,970 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:03,978 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:03,978 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:03,978 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:56:03,978 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. started... 2023-04-08 20:56:03,978 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 06f3cb8ce0404e826c51a372e2c9bfbf 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-08 20:56:03,979 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=A 2023-04-08 20:56:03,979 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:03,979 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=B 2023-04-08 20:56:03,979 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:03,979 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=C 2023-04-08 20:56:03,979 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:03,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:56:03,988 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. as already flushing 2023-04-08 20:56:04,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:04,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41716 deadline: 1680987424015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:04,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:04,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41660 deadline: 1680987424015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:04,123 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:04,124 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:04,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41716 deadline: 1680987424120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:04,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41660 deadline: 1680987424120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:04,326 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:04,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41716 deadline: 1680987424325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:04,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:04,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41660 deadline: 1680987424326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:04,415 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304085fc09247c3b546d88a4a89b08bde74e1 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304085fc09247c3b546d88a4a89b08bde74e1 2023-04-08 20:56:04,518 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/7834edb0185443aa9659a51deb915667 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/7834edb0185443aa9659a51deb915667 2023-04-08 20:56:04,524 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 06f3cb8ce0404e826c51a372e2c9bfbf/A of 06f3cb8ce0404e826c51a372e2c9bfbf into 7834edb0185443aa9659a51deb915667(size=26.2 K), total size for store is 51.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:56:04,524 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:56:04,524 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf., storeName=06f3cb8ce0404e826c51a372e2c9bfbf/A, priority=10, startTime=1680987363589; duration=0sec 2023-04-08 20:56:04,524 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:56:04,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:04,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41716 deadline: 1680987424628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:04,632 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:04,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41660 deadline: 1680987424630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:04,662 DEBUG [Thread-983] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6683e622 to 127.0.0.1:52440 2023-04-08 20:56:04,662 DEBUG [Thread-983] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:56:04,664 DEBUG [Thread-987] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4890ce53 to 127.0.0.1:52440 2023-04-08 20:56:04,664 DEBUG [Thread-987] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:56:04,665 DEBUG [Thread-981] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34b669b1 to 127.0.0.1:52440 2023-04-08 20:56:04,665 DEBUG [Thread-981] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:56:04,666 DEBUG [Thread-985] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x615d8f34 to 127.0.0.1:52440 2023-04-08 20:56:04,666 DEBUG [Thread-985] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:56:04,820 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=373, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/c650ab6e5a6d427987728fe8ee7e2658 2023-04-08 20:56:04,839 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=373 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/f435dac4abaa4258883803eca507776c 2023-04-08 20:56:05,134 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:05,134 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:05,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41660 deadline: 1680987425134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:05,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41716 deadline: 1680987425134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:05,260 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=373 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/71fc83283e8b4270974029d7ef37ad01 2023-04-08 20:56:05,265 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/c650ab6e5a6d427987728fe8ee7e2658 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/c650ab6e5a6d427987728fe8ee7e2658 2023-04-08 20:56:05,269 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/c650ab6e5a6d427987728fe8ee7e2658, entries=150, sequenceid=373, filesize=25.4 K 2023-04-08 20:56:05,270 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/f435dac4abaa4258883803eca507776c as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/f435dac4abaa4258883803eca507776c 2023-04-08 20:56:05,274 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/f435dac4abaa4258883803eca507776c, entries=150, sequenceid=373, filesize=11.9 K 2023-04-08 20:56:05,274 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/71fc83283e8b4270974029d7ef37ad01 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/71fc83283e8b4270974029d7ef37ad01 2023-04-08 20:56:05,279 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/71fc83283e8b4270974029d7ef37ad01, entries=150, sequenceid=373, filesize=11.9 K 2023-04-08 20:56:05,280 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 06f3cb8ce0404e826c51a372e2c9bfbf in 1302ms, sequenceid=373, compaction requested=true 2023-04-08 20:56:05,280 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:56:05,280 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:56:05,280 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:05,280 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:05,280 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:05,280 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:05,280 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:05,281 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:05,281 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:05,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:05,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:05,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:05,282 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:05,282 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:05,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:05,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:05,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:05,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:05,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:05,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:05,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5da7b781[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:05,283 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:05,283 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:05,284 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:05,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:05,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:05,284 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:05,284 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:05,284 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:05,284 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:05,284 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:05,285 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:05,285 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:05,285 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:05,285 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:05,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:05,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:05,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:05,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:05,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:05,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:05,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:05,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:05,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:05,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:05,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:05,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:05,287 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:05,287 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:05,287 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:05,287 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:05,287 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:05,290 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:05,292 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:05,292 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:05,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:05,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:05,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:05,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:05,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:05,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:05,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:05,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:05,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:05,295 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:05,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:05,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:05,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:05,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:05,296 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:05,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:05,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:05,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:05,302 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:05,302 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:05,302 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:05,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:05,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:05,302 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:05,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:05,302 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:05,302 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:05,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:05,302 DEBUG [Thread-980] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:05,302 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:05,302 DEBUG [Thread-980] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:05,302 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:05,302 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:05,302 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:05,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:05,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:05,402 DEBUG [Thread-980] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:05,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:06,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:56:06,138 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 06f3cb8ce0404e826c51a372e2c9bfbf 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-08 20:56:06,139 DEBUG [Thread-976] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4af56759 to 127.0.0.1:52440 2023-04-08 20:56:06,139 DEBUG [Thread-976] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:56:06,139 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=A 2023-04-08 20:56:06,139 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:06,139 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=B 2023-04-08 20:56:06,139 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:06,139 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=C 2023-04-08 20:56:06,139 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:06,143 DEBUG [Thread-978] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5d07b2f5 to 127.0.0.1:52440 2023-04-08 20:56:06,143 DEBUG [Thread-978] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:56:06,154 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304082cb486fd577347819386b0fee91e3955 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304082cb486fd577347819386b0fee91e3955 2023-04-08 20:56:06,565 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=390, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/464d5f4774284b12ada2e29de6103d18 2023-04-08 20:56:06,758 DEBUG [Thread-970] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6dc16927 to 127.0.0.1:52440 2023-04-08 20:56:06,758 DEBUG [Thread-970] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:56:06,777 DEBUG [Thread-972] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x029d68f7 to 127.0.0.1:52440 2023-04-08 20:56:06,777 DEBUG [Thread-972] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:56:06,980 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=390 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/71b8b5f249264a0fbdea96e62df7c3ec 2023-04-08 20:56:07,392 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=390 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/e81653d73b3b4b03b13823e190e593d4 2023-04-08 20:56:07,403 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/464d5f4774284b12ada2e29de6103d18 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/464d5f4774284b12ada2e29de6103d18 2023-04-08 20:56:07,408 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/464d5f4774284b12ada2e29de6103d18, entries=150, sequenceid=390, filesize=25.4 K 2023-04-08 20:56:07,409 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/71b8b5f249264a0fbdea96e62df7c3ec as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/71b8b5f249264a0fbdea96e62df7c3ec 2023-04-08 20:56:07,415 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/71b8b5f249264a0fbdea96e62df7c3ec, entries=150, sequenceid=390, filesize=11.9 K 2023-04-08 20:56:07,416 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/e81653d73b3b4b03b13823e190e593d4 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/e81653d73b3b4b03b13823e190e593d4 2023-04-08 20:56:07,423 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/e81653d73b3b4b03b13823e190e593d4, entries=150, sequenceid=390, filesize=11.9 K 2023-04-08 20:56:07,424 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=20.13 KB/20610 for 06f3cb8ce0404e826c51a372e2c9bfbf in 1286ms, sequenceid=390, compaction requested=true 2023-04-08 20:56:07,424 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:56:07,424 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:56:07,424 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 20:56:07,424 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:56:07,424 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:56:07,424 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 20:56:07,426 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104968 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 20:56:07,426 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): 06f3cb8ce0404e826c51a372e2c9bfbf/A is initiating minor compaction (all files) 2023-04-08 20:56:07,426 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49528 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 20:56:07,426 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 06f3cb8ce0404e826c51a372e2c9bfbf/A in TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:56:07,426 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 06f3cb8ce0404e826c51a372e2c9bfbf/B is initiating minor compaction (all files) 2023-04-08 20:56:07,427 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/7834edb0185443aa9659a51deb915667, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/63367cea9b4d463f845f664ab323ddcb, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/c650ab6e5a6d427987728fe8ee7e2658, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/464d5f4774284b12ada2e29de6103d18] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp, totalSize=102.5 K 2023-04-08 20:56:07,427 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 06f3cb8ce0404e826c51a372e2c9bfbf/B in TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:56:07,427 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/d46cf04703a3412db7f81592ed55faa1, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/cb16e8a3fe9542b08c34270ba5cb4f45, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/f435dac4abaa4258883803eca507776c, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/71b8b5f249264a0fbdea96e62df7c3ec] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp, totalSize=48.4 K 2023-04-08 20:56:07,427 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 7834edb0185443aa9659a51deb915667, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1680987362373 2023-04-08 20:56:07,427 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting d46cf04703a3412db7f81592ed55faa1, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1680987362373 2023-04-08 20:56:07,428 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 63367cea9b4d463f845f664ab323ddcb, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=354, earliestPutTs=1680987363538 2023-04-08 20:56:07,428 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting cb16e8a3fe9542b08c34270ba5cb4f45, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=354, earliestPutTs=1680987363538 2023-04-08 20:56:07,428 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting c650ab6e5a6d427987728fe8ee7e2658, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1680987363669 2023-04-08 20:56:07,428 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting f435dac4abaa4258883803eca507776c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1680987363669 2023-04-08 20:56:07,428 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 464d5f4774284b12ada2e29de6103d18, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1680987363997 2023-04-08 20:56:07,429 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 71b8b5f249264a0fbdea96e62df7c3ec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1680987363997 2023-04-08 20:56:07,442 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 06f3cb8ce0404e826c51a372e2c9bfbf#B#compaction#207 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:56:07,445 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 06f3cb8ce0404e826c51a372e2c9bfbf#A#compaction#208 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:56:07,866 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/2afef730af0b4c6a84f16d70e2c87b2a as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/2afef730af0b4c6a84f16d70e2c87b2a 2023-04-08 20:56:07,868 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/be5bce2c43034d118d2e9d83a94bf971 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/be5bce2c43034d118d2e9d83a94bf971 2023-04-08 20:56:07,870 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 06f3cb8ce0404e826c51a372e2c9bfbf/B of 06f3cb8ce0404e826c51a372e2c9bfbf into 2afef730af0b4c6a84f16d70e2c87b2a(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:56:07,870 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:56:07,871 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf., storeName=06f3cb8ce0404e826c51a372e2c9bfbf/B, priority=12, startTime=1680987367424; duration=0sec 2023-04-08 20:56:07,871 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:56:07,871 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 20:56:07,872 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49528 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 20:56:07,872 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 06f3cb8ce0404e826c51a372e2c9bfbf/C is initiating minor compaction (all files) 2023-04-08 20:56:07,872 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 06f3cb8ce0404e826c51a372e2c9bfbf/C in TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:56:07,872 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/63da166db7f84fd788ca9c4760bba13a, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/1778cacb9c70493897fd84b968d5427e, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/71fc83283e8b4270974029d7ef37ad01, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/e81653d73b3b4b03b13823e190e593d4] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp, totalSize=48.4 K 2023-04-08 20:56:07,873 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 63da166db7f84fd788ca9c4760bba13a, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1680987362373 2023-04-08 20:56:07,873 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 1778cacb9c70493897fd84b968d5427e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=354, earliestPutTs=1680987363538 2023-04-08 20:56:07,873 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 06f3cb8ce0404e826c51a372e2c9bfbf/A of 06f3cb8ce0404e826c51a372e2c9bfbf into be5bce2c43034d118d2e9d83a94bf971(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:56:07,873 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:56:07,873 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf., storeName=06f3cb8ce0404e826c51a372e2c9bfbf/A, priority=12, startTime=1680987367424; duration=0sec 2023-04-08 20:56:07,873 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 71fc83283e8b4270974029d7ef37ad01, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1680987363669 2023-04-08 20:56:07,873 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:56:07,873 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting e81653d73b3b4b03b13823e190e593d4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1680987363997 2023-04-08 20:56:07,880 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 06f3cb8ce0404e826c51a372e2c9bfbf#C#compaction#209 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:56:08,295 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/46327c80a8ec4f25be367867e712899e as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/46327c80a8ec4f25be367867e712899e 2023-04-08 20:56:08,304 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 06f3cb8ce0404e826c51a372e2c9bfbf/C of 06f3cb8ce0404e826c51a372e2c9bfbf into 46327c80a8ec4f25be367867e712899e(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:56:08,304 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:56:08,304 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf., storeName=06f3cb8ce0404e826c51a372e2c9bfbf/C, priority=12, startTime=1680987367424; duration=0sec 2023-04-08 20:56:08,304 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:56:13,115 DEBUG [Thread-974] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6322d500 to 127.0.0.1:52440 2023-04-08 20:56:13,115 DEBUG [Thread-974] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:56:13,115 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-08 20:56:13,115 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(394): wrote 20 2023-04-08 20:56:13,115 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(394): wrote 25 2023-04-08 20:56:13,116 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(394): wrote 94 2023-04-08 20:56:13,116 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(394): wrote 91 2023-04-08 20:56:13,116 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(394): wrote 79 2023-04-08 20:56:13,116 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-08 20:56:13,116 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(398): read 5362 2023-04-08 20:56:13,116 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(398): read 5346 2023-04-08 20:56:13,116 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-08 20:56:13,116 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(402): scanned 2293 2023-04-08 20:56:13,116 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(403): verified 6878 rows 2023-04-08 20:56:13,116 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(402): scanned 2293 2023-04-08 20:56:13,116 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(403): verified 6878 rows 2023-04-08 20:56:13,116 INFO [Listener at localhost.localdomain/41823] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-08 20:56:13,116 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10dd04e6 to 127.0.0.1:52440 2023-04-08 20:56:13,116 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:56:13,117 INFO [Listener at localhost.localdomain/41823] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-08 20:56:13,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-08 20:56:13,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-08 20:56:13,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-08 20:56:13,126 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680987373126"}]},"ts":"1680987373126"} 2023-04-08 20:56:13,127 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-08 20:56:13,130 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-08 20:56:13,131 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=06f3cb8ce0404e826c51a372e2c9bfbf, UNASSIGN}] 2023-04-08 20:56:13,133 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=06f3cb8ce0404e826c51a372e2c9bfbf, UNASSIGN 2023-04-08 20:56:13,134 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=06f3cb8ce0404e826c51a372e2c9bfbf, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:13,134 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987373133"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680987373133"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680987373133"}]},"ts":"1680987373133"} 2023-04-08 20:56:13,135 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601}] 2023-04-08 20:56:13,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-08 20:56:13,288 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:56:13,288 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 06f3cb8ce0404e826c51a372e2c9bfbf, disabling compactions & flushes 2023-04-08 20:56:13,288 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:56:13,288 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:56:13,288 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. after waiting 0 ms 2023-04-08 20:56:13,288 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:56:13,288 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 06f3cb8ce0404e826c51a372e2c9bfbf 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-08 20:56:13,288 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=A 2023-04-08 20:56:13,288 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:13,288 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=B 2023-04-08 20:56:13,288 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:13,288 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 06f3cb8ce0404e826c51a372e2c9bfbf, store=C 2023-04-08 20:56:13,288 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:13,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-08 20:56:13,699 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408ac66b615ad4b4ed496a35cd7f6ccbb20 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408ac66b615ad4b4ed496a35cd7f6ccbb20 2023-04-08 20:56:13,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-08 20:56:14,105 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=400, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/27ab50d93cfe416bb557e91b23fad337 2023-04-08 20:56:14,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-08 20:56:14,523 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/263d45cfe47843a698655fff843db69c 2023-04-08 20:56:14,778 DEBUG [regionserver/jenkins-hbase20:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-08 20:56:14,933 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/5747ca8a2658430eb4f15f06f8c5504b 2023-04-08 20:56:14,937 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/A/27ab50d93cfe416bb557e91b23fad337 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/27ab50d93cfe416bb557e91b23fad337 2023-04-08 20:56:14,940 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/27ab50d93cfe416bb557e91b23fad337, entries=150, sequenceid=400, filesize=25.4 K 2023-04-08 20:56:14,941 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/B/263d45cfe47843a698655fff843db69c as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/263d45cfe47843a698655fff843db69c 2023-04-08 20:56:14,944 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/263d45cfe47843a698655fff843db69c, entries=150, sequenceid=400, filesize=11.9 K 2023-04-08 20:56:14,945 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/.tmp/C/5747ca8a2658430eb4f15f06f8c5504b as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/5747ca8a2658430eb4f15f06f8c5504b 2023-04-08 20:56:14,948 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/5747ca8a2658430eb4f15f06f8c5504b, entries=150, sequenceid=400, filesize=11.9 K 2023-04-08 20:56:14,948 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 06f3cb8ce0404e826c51a372e2c9bfbf in 1660ms, sequenceid=400, compaction requested=false 2023-04-08 20:56:14,964 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/3c80dc546b57474aa05130339c354eda, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/ee155265376b4efeb506b8e75a8cc6ea, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/5d1daf7ff7e342c7bb963a46f026290a, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/1aa088522a9b4a9b8ee41e6991f99bc0, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/8fa6b0e2151c4c7083391d5a41bce683, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/2a795156e07241f79d5f03b5453e8a47, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/ad3bd7f330584292bbaf3d629f0ba73c, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/9495de42f4874f9582274e9540fd3283, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/2344f51dbb5a4b638bc3098e191b9909, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/a761fe9ad79a430f837a670750aa2bad, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/6bbba4261e0c425392e0c0bc40c4c1f1, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/7dbb9acbed244341ab2bb8faaa27ef77, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/578832365cb54473a44b90ef332213dd, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/100bd58e18394ee2ae96f06fb8026f62, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/28fa970266d447469c0993fc2bfc39c5, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/440a642599474ce58f08fccddfee3853, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/d992eb4846bb4f629e9fee232b2e62f3, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/17ce95c063f84bf4aee8d2d53d245cb2, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/f42c11af18ec4e81934f4f776bd4882f, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/22142b4928624077a087e984e03db47c, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/104af472bfa745858ad76c82328bc062, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/c3c91574c83a40c5933063cabf762db2, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/e20fac8da609434d8341cf0084a04c71, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/7834edb0185443aa9659a51deb915667, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/63367cea9b4d463f845f664ab323ddcb, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/c650ab6e5a6d427987728fe8ee7e2658, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/464d5f4774284b12ada2e29de6103d18] to archive 2023-04-08 20:56:14,966 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 20:56:14,967 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/3c80dc546b57474aa05130339c354eda to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/3c80dc546b57474aa05130339c354eda 2023-04-08 20:56:14,968 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/ee155265376b4efeb506b8e75a8cc6ea to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/ee155265376b4efeb506b8e75a8cc6ea 2023-04-08 20:56:14,969 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/5d1daf7ff7e342c7bb963a46f026290a to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/5d1daf7ff7e342c7bb963a46f026290a 2023-04-08 20:56:14,971 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/1aa088522a9b4a9b8ee41e6991f99bc0 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/1aa088522a9b4a9b8ee41e6991f99bc0 2023-04-08 20:56:14,972 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/8fa6b0e2151c4c7083391d5a41bce683 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/8fa6b0e2151c4c7083391d5a41bce683 2023-04-08 20:56:14,973 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/2a795156e07241f79d5f03b5453e8a47 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/2a795156e07241f79d5f03b5453e8a47 2023-04-08 20:56:14,974 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/ad3bd7f330584292bbaf3d629f0ba73c to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/ad3bd7f330584292bbaf3d629f0ba73c 2023-04-08 20:56:14,975 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/9495de42f4874f9582274e9540fd3283 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/9495de42f4874f9582274e9540fd3283 2023-04-08 20:56:14,976 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/2344f51dbb5a4b638bc3098e191b9909 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/2344f51dbb5a4b638bc3098e191b9909 2023-04-08 20:56:14,977 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/a761fe9ad79a430f837a670750aa2bad to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/a761fe9ad79a430f837a670750aa2bad 2023-04-08 20:56:14,978 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/6bbba4261e0c425392e0c0bc40c4c1f1 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/6bbba4261e0c425392e0c0bc40c4c1f1 2023-04-08 20:56:14,979 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/7dbb9acbed244341ab2bb8faaa27ef77 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/7dbb9acbed244341ab2bb8faaa27ef77 2023-04-08 20:56:14,980 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/578832365cb54473a44b90ef332213dd to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/578832365cb54473a44b90ef332213dd 2023-04-08 20:56:14,982 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/100bd58e18394ee2ae96f06fb8026f62 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/100bd58e18394ee2ae96f06fb8026f62 2023-04-08 20:56:14,983 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/28fa970266d447469c0993fc2bfc39c5 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/28fa970266d447469c0993fc2bfc39c5 2023-04-08 20:56:14,984 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/440a642599474ce58f08fccddfee3853 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/440a642599474ce58f08fccddfee3853 2023-04-08 20:56:14,985 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/d992eb4846bb4f629e9fee232b2e62f3 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/d992eb4846bb4f629e9fee232b2e62f3 2023-04-08 20:56:14,986 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/17ce95c063f84bf4aee8d2d53d245cb2 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/17ce95c063f84bf4aee8d2d53d245cb2 2023-04-08 20:56:14,987 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/f42c11af18ec4e81934f4f776bd4882f to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/f42c11af18ec4e81934f4f776bd4882f 2023-04-08 20:56:14,988 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/22142b4928624077a087e984e03db47c to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/22142b4928624077a087e984e03db47c 2023-04-08 20:56:14,989 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/104af472bfa745858ad76c82328bc062 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/104af472bfa745858ad76c82328bc062 2023-04-08 20:56:14,990 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/c3c91574c83a40c5933063cabf762db2 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/c3c91574c83a40c5933063cabf762db2 2023-04-08 20:56:14,991 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/e20fac8da609434d8341cf0084a04c71 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/e20fac8da609434d8341cf0084a04c71 2023-04-08 20:56:14,992 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/7834edb0185443aa9659a51deb915667 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/7834edb0185443aa9659a51deb915667 2023-04-08 20:56:14,993 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/63367cea9b4d463f845f664ab323ddcb to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/63367cea9b4d463f845f664ab323ddcb 2023-04-08 20:56:14,994 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/c650ab6e5a6d427987728fe8ee7e2658 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/c650ab6e5a6d427987728fe8ee7e2658 2023-04-08 20:56:14,995 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/464d5f4774284b12ada2e29de6103d18 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/464d5f4774284b12ada2e29de6103d18 2023-04-08 20:56:14,998 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/5f043b692e3746d5a41bfd2c35c6d92d, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/7e5c7cb7ff9c4d95aa2a078d83da8169, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/3d0d5f5c72d94f948d54abda75da8b56, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/d604e75322fd4d40ba25ac3442a6d840, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/3aa6663032f04c82bbac2b5af2e55ff1, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/413c7d407f534a6babd50042971f7427, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/4a266c949449432d87425df1d5213b3c, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/ae54f2a3192f490db7cb70258c4cda70, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/847561cdaf9142faadaa3fa9708abf37, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/4cc3f4af84614958b75109bc42c42070, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/7c3b41df8847405ba6bfad50853493c9, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/adb92f0f261044799a9cb438236b1006, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/530573f8c3424874b0be4107dfaa1026, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/2c6107af49e84ec0a53b302bc308eaaa, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/dff302b9688d46018c7274056fa7b3fc, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/3177cea219784386899f0e090c51811f, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/5abb39f4e41d48f984183f3a91cfa414, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/354f83ccde8644eeaeda4bf6d9a6698b, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/9631714bc9e54b658a95e0ba074e2c4a, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/a520694bf2844712b9fb0b4352f3aa0d, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/80b90c0344594afa9c09f37bd0983fcb, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/d02f19c1dd2d431cbc78f26be161a464, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/d46cf04703a3412db7f81592ed55faa1, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/e0e6c4c7b8f3410b93eb87b9be62dd66, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/cb16e8a3fe9542b08c34270ba5cb4f45, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/f435dac4abaa4258883803eca507776c, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/71b8b5f249264a0fbdea96e62df7c3ec] to archive 2023-04-08 20:56:14,999 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 20:56:15,000 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/5f043b692e3746d5a41bfd2c35c6d92d to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/5f043b692e3746d5a41bfd2c35c6d92d 2023-04-08 20:56:15,001 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/7e5c7cb7ff9c4d95aa2a078d83da8169 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/7e5c7cb7ff9c4d95aa2a078d83da8169 2023-04-08 20:56:15,002 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/3d0d5f5c72d94f948d54abda75da8b56 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/3d0d5f5c72d94f948d54abda75da8b56 2023-04-08 20:56:15,003 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/d604e75322fd4d40ba25ac3442a6d840 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/d604e75322fd4d40ba25ac3442a6d840 2023-04-08 20:56:15,004 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/3aa6663032f04c82bbac2b5af2e55ff1 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/3aa6663032f04c82bbac2b5af2e55ff1 2023-04-08 20:56:15,005 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/413c7d407f534a6babd50042971f7427 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/413c7d407f534a6babd50042971f7427 2023-04-08 20:56:15,006 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/4a266c949449432d87425df1d5213b3c to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/4a266c949449432d87425df1d5213b3c 2023-04-08 20:56:15,007 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/ae54f2a3192f490db7cb70258c4cda70 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/ae54f2a3192f490db7cb70258c4cda70 2023-04-08 20:56:15,008 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/847561cdaf9142faadaa3fa9708abf37 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/847561cdaf9142faadaa3fa9708abf37 2023-04-08 20:56:15,009 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/4cc3f4af84614958b75109bc42c42070 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/4cc3f4af84614958b75109bc42c42070 2023-04-08 20:56:15,010 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/7c3b41df8847405ba6bfad50853493c9 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/7c3b41df8847405ba6bfad50853493c9 2023-04-08 20:56:15,011 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/adb92f0f261044799a9cb438236b1006 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/adb92f0f261044799a9cb438236b1006 2023-04-08 20:56:15,012 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/530573f8c3424874b0be4107dfaa1026 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/530573f8c3424874b0be4107dfaa1026 2023-04-08 20:56:15,013 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/2c6107af49e84ec0a53b302bc308eaaa to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/2c6107af49e84ec0a53b302bc308eaaa 2023-04-08 20:56:15,014 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/dff302b9688d46018c7274056fa7b3fc to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/dff302b9688d46018c7274056fa7b3fc 2023-04-08 20:56:15,015 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/3177cea219784386899f0e090c51811f to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/3177cea219784386899f0e090c51811f 2023-04-08 20:56:15,016 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/5abb39f4e41d48f984183f3a91cfa414 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/5abb39f4e41d48f984183f3a91cfa414 2023-04-08 20:56:15,017 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/354f83ccde8644eeaeda4bf6d9a6698b to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/354f83ccde8644eeaeda4bf6d9a6698b 2023-04-08 20:56:15,018 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/9631714bc9e54b658a95e0ba074e2c4a to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/9631714bc9e54b658a95e0ba074e2c4a 2023-04-08 20:56:15,019 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/a520694bf2844712b9fb0b4352f3aa0d to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/a520694bf2844712b9fb0b4352f3aa0d 2023-04-08 20:56:15,020 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/80b90c0344594afa9c09f37bd0983fcb to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/80b90c0344594afa9c09f37bd0983fcb 2023-04-08 20:56:15,020 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/d02f19c1dd2d431cbc78f26be161a464 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/d02f19c1dd2d431cbc78f26be161a464 2023-04-08 20:56:15,021 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/d46cf04703a3412db7f81592ed55faa1 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/d46cf04703a3412db7f81592ed55faa1 2023-04-08 20:56:15,022 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/e0e6c4c7b8f3410b93eb87b9be62dd66 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/e0e6c4c7b8f3410b93eb87b9be62dd66 2023-04-08 20:56:15,023 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/cb16e8a3fe9542b08c34270ba5cb4f45 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/cb16e8a3fe9542b08c34270ba5cb4f45 2023-04-08 20:56:15,024 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/f435dac4abaa4258883803eca507776c to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/f435dac4abaa4258883803eca507776c 2023-04-08 20:56:15,025 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/71b8b5f249264a0fbdea96e62df7c3ec to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/71b8b5f249264a0fbdea96e62df7c3ec 2023-04-08 20:56:15,028 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/205c1805d57d43dca4d0936321ac0af2, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/44db0d860553479cb9c80f7657e26893, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/3aa68c08ba2d42819d5d6a628be27587, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/458e8ca2827b4b76b5ce28a7399063e6, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/46c918bff8a94883a0e2ef988d39c719, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/b2f920cf84f24135a0eec58cc131ff29, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/398b7bfd810e45ea8452569d63656073, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/fe2b9554e18b4d6ca180c3af45a12a91, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/d411662c97f140608a40c07b815145b7, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/9248ccf6efa242689421db2bd960dd9a, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/2c529efc887447a28c97802955d6c4ea, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/4cab6128916f459c891cd3ccea997e03, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/84512de0fcc9403c8d3e92a23c941080, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/988044634b4b4043860fd656ead62259, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/a4f0e0e07e014f029c0f9b4f6b5174da, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/8da4688e0a8a431c923e530cc6fdfb99, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/5bb49fe6b3ee46aca0084d2f6b76be45, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/dfad5af107d24fe995ef37d0e54d53d5, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/727e8e1bd59e4890b28e00388ce84f5f, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/83d23b3108f14d23a0a5bd50f88062d8, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/db170529132a4c46828f0a957edfacf6, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/406ebedd98d348ef956f530ba005846c, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/63da166db7f84fd788ca9c4760bba13a, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/52698dad24f14bf6be66fc201fabc0c4, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/1778cacb9c70493897fd84b968d5427e, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/71fc83283e8b4270974029d7ef37ad01, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/e81653d73b3b4b03b13823e190e593d4] to archive 2023-04-08 20:56:15,029 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 20:56:15,030 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/205c1805d57d43dca4d0936321ac0af2 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/205c1805d57d43dca4d0936321ac0af2 2023-04-08 20:56:15,031 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/44db0d860553479cb9c80f7657e26893 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/44db0d860553479cb9c80f7657e26893 2023-04-08 20:56:15,032 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/3aa68c08ba2d42819d5d6a628be27587 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/3aa68c08ba2d42819d5d6a628be27587 2023-04-08 20:56:15,033 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/458e8ca2827b4b76b5ce28a7399063e6 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/458e8ca2827b4b76b5ce28a7399063e6 2023-04-08 20:56:15,034 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/46c918bff8a94883a0e2ef988d39c719 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/46c918bff8a94883a0e2ef988d39c719 2023-04-08 20:56:15,035 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/b2f920cf84f24135a0eec58cc131ff29 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/b2f920cf84f24135a0eec58cc131ff29 2023-04-08 20:56:15,036 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/398b7bfd810e45ea8452569d63656073 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/398b7bfd810e45ea8452569d63656073 2023-04-08 20:56:15,037 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/fe2b9554e18b4d6ca180c3af45a12a91 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/fe2b9554e18b4d6ca180c3af45a12a91 2023-04-08 20:56:15,038 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/d411662c97f140608a40c07b815145b7 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/d411662c97f140608a40c07b815145b7 2023-04-08 20:56:15,039 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/9248ccf6efa242689421db2bd960dd9a to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/9248ccf6efa242689421db2bd960dd9a 2023-04-08 20:56:15,040 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/2c529efc887447a28c97802955d6c4ea to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/2c529efc887447a28c97802955d6c4ea 2023-04-08 20:56:15,041 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/4cab6128916f459c891cd3ccea997e03 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/4cab6128916f459c891cd3ccea997e03 2023-04-08 20:56:15,042 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/84512de0fcc9403c8d3e92a23c941080 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/84512de0fcc9403c8d3e92a23c941080 2023-04-08 20:56:15,043 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/988044634b4b4043860fd656ead62259 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/988044634b4b4043860fd656ead62259 2023-04-08 20:56:15,044 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/a4f0e0e07e014f029c0f9b4f6b5174da to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/a4f0e0e07e014f029c0f9b4f6b5174da 2023-04-08 20:56:15,045 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/8da4688e0a8a431c923e530cc6fdfb99 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/8da4688e0a8a431c923e530cc6fdfb99 2023-04-08 20:56:15,046 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/5bb49fe6b3ee46aca0084d2f6b76be45 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/5bb49fe6b3ee46aca0084d2f6b76be45 2023-04-08 20:56:15,047 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/dfad5af107d24fe995ef37d0e54d53d5 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/dfad5af107d24fe995ef37d0e54d53d5 2023-04-08 20:56:15,048 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/727e8e1bd59e4890b28e00388ce84f5f to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/727e8e1bd59e4890b28e00388ce84f5f 2023-04-08 20:56:15,050 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/83d23b3108f14d23a0a5bd50f88062d8 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/83d23b3108f14d23a0a5bd50f88062d8 2023-04-08 20:56:15,051 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/db170529132a4c46828f0a957edfacf6 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/db170529132a4c46828f0a957edfacf6 2023-04-08 20:56:15,052 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/406ebedd98d348ef956f530ba005846c to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/406ebedd98d348ef956f530ba005846c 2023-04-08 20:56:15,053 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/63da166db7f84fd788ca9c4760bba13a to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/63da166db7f84fd788ca9c4760bba13a 2023-04-08 20:56:15,054 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/52698dad24f14bf6be66fc201fabc0c4 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/52698dad24f14bf6be66fc201fabc0c4 2023-04-08 20:56:15,056 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/1778cacb9c70493897fd84b968d5427e to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/1778cacb9c70493897fd84b968d5427e 2023-04-08 20:56:15,057 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/71fc83283e8b4270974029d7ef37ad01 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/71fc83283e8b4270974029d7ef37ad01 2023-04-08 20:56:15,058 DEBUG [StoreCloser-TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/e81653d73b3b4b03b13823e190e593d4 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/e81653d73b3b4b03b13823e190e593d4 2023-04-08 20:56:15,063 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/recovered.edits/403.seqid, newMaxSeqId=403, maxSeqId=4 2023-04-08 20:56:15,065 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf. 2023-04-08 20:56:15,065 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 06f3cb8ce0404e826c51a372e2c9bfbf: 2023-04-08 20:56:15,066 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:56:15,067 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=06f3cb8ce0404e826c51a372e2c9bfbf, regionState=CLOSED 2023-04-08 20:56:15,067 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987375067"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680987375067"}]},"ts":"1680987375067"} 2023-04-08 20:56:15,069 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-08 20:56:15,069 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 06f3cb8ce0404e826c51a372e2c9bfbf, server=jenkins-hbase20.apache.org,41091,1680987313601 in 1.9330 sec 2023-04-08 20:56:15,071 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-08 20:56:15,071 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=06f3cb8ce0404e826c51a372e2c9bfbf, UNASSIGN in 1.9380 sec 2023-04-08 20:56:15,071 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680987375071"}]},"ts":"1680987375071"} 2023-04-08 20:56:15,072 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-08 20:56:15,073 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-08 20:56:15,075 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9560 sec 2023-04-08 20:56:15,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-08 20:56:15,232 INFO [Listener at localhost.localdomain/41823] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-08 20:56:15,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-08 20:56:15,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 20:56:15,234 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 20:56:15,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-08 20:56:15,235 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 20:56:15,239 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:56:15,242 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A, FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B, FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C, FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/recovered.edits] 2023-04-08 20:56:15,244 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/27ab50d93cfe416bb557e91b23fad337 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/27ab50d93cfe416bb557e91b23fad337 2023-04-08 20:56:15,245 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/be5bce2c43034d118d2e9d83a94bf971 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/A/be5bce2c43034d118d2e9d83a94bf971 2023-04-08 20:56:15,247 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/263d45cfe47843a698655fff843db69c to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/263d45cfe47843a698655fff843db69c 2023-04-08 20:56:15,248 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/2afef730af0b4c6a84f16d70e2c87b2a to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/B/2afef730af0b4c6a84f16d70e2c87b2a 2023-04-08 20:56:15,250 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/46327c80a8ec4f25be367867e712899e to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/46327c80a8ec4f25be367867e712899e 2023-04-08 20:56:15,251 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/5747ca8a2658430eb4f15f06f8c5504b to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/C/5747ca8a2658430eb4f15f06f8c5504b 2023-04-08 20:56:15,253 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/recovered.edits/403.seqid to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf/recovered.edits/403.seqid 2023-04-08 20:56:15,254 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/06f3cb8ce0404e826c51a372e2c9bfbf 2023-04-08 20:56:15,254 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-08 20:56:15,254 DEBUG [PEWorker-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-08 20:56:15,255 DEBUG [PEWorker-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-08 20:56:15,259 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040801b2e4d436f643279f975560d655e160 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040801b2e4d436f643279f975560d655e160 2023-04-08 20:56:15,261 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408148ae4b529f4448096016315acbd5180 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408148ae4b529f4448096016315acbd5180 2023-04-08 20:56:15,262 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304082cb486fd577347819386b0fee91e3955 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304082cb486fd577347819386b0fee91e3955 2023-04-08 20:56:15,264 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408303335c41c164086bf55a70f3320bc46 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408303335c41c164086bf55a70f3320bc46 2023-04-08 20:56:15,266 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304083f82879b45a54ca88282e27508bf1543 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304083f82879b45a54ca88282e27508bf1543 2023-04-08 20:56:15,267 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304085fc09247c3b546d88a4a89b08bde74e1 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304085fc09247c3b546d88a4a89b08bde74e1 2023-04-08 20:56:15,268 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040869863ac5d402407b86920e7e361fb288 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040869863ac5d402407b86920e7e361fb288 2023-04-08 20:56:15,269 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040869bf602fd5484309b01e88982a9d8e00 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040869bf602fd5484309b01e88982a9d8e00 2023-04-08 20:56:15,270 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304086b4b887829a44649a5796127e27f7f8a to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304086b4b887829a44649a5796127e27f7f8a 2023-04-08 20:56:15,272 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304086cb0203405294211b8725611bc445543 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304086cb0203405294211b8725611bc445543 2023-04-08 20:56:15,273 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304086dc6a579164346e397ed12047296137a to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304086dc6a579164346e397ed12047296137a 2023-04-08 20:56:15,274 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408725834a0badf4803b03584f4bc4e1622 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408725834a0badf4803b03584f4bc4e1622 2023-04-08 20:56:15,275 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040872a055dd7faf4a56aface8c527242cb0 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040872a055dd7faf4a56aface8c527242cb0 2023-04-08 20:56:15,276 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408932f1630221e4412acc5adee6c97e1b2 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408932f1630221e4412acc5adee6c97e1b2 2023-04-08 20:56:15,278 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040896a263d4a02b44158aefe2c0ab6fe65e to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040896a263d4a02b44158aefe2c0ab6fe65e 2023-04-08 20:56:15,279 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304089728685f37474969b6e3edf3fa742526 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304089728685f37474969b6e3edf3fa742526 2023-04-08 20:56:15,280 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408a826f5e2e4b04cc8ad3d620aa22d691b to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408a826f5e2e4b04cc8ad3d620aa22d691b 2023-04-08 20:56:15,281 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408a916e664419c41319ad436c54ef4828d to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408a916e664419c41319ad436c54ef4828d 2023-04-08 20:56:15,282 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408ac66b615ad4b4ed496a35cd7f6ccbb20 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408ac66b615ad4b4ed496a35cd7f6ccbb20 2023-04-08 20:56:15,283 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408bc54aa4804974b3abff8daf34ce62924 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408bc54aa4804974b3abff8daf34ce62924 2023-04-08 20:56:15,284 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408cdb2a11277b5471bbe38c3e98c783f8e to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408cdb2a11277b5471bbe38c3e98c783f8e 2023-04-08 20:56:15,285 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408dfa084f1d560421485fa0e46b4c7f1f2 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408dfa084f1d560421485fa0e46b4c7f1f2 2023-04-08 20:56:15,286 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408fbd97c5eceac4b07ae15c42566393cca to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408fbd97c5eceac4b07ae15c42566393cca 2023-04-08 20:56:15,286 DEBUG [PEWorker-4] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-08 20:56:15,288 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 20:56:15,290 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-08 20:56:15,296 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-08 20:56:15,297 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 20:56:15,297 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-08 20:56:15,297 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680987375297"}]},"ts":"9223372036854775807"} 2023-04-08 20:56:15,299 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-08 20:56:15,299 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 06f3cb8ce0404e826c51a372e2c9bfbf, NAME => 'TestAcidGuarantees,,1680987341873.06f3cb8ce0404e826c51a372e2c9bfbf.', STARTKEY => '', ENDKEY => ''}] 2023-04-08 20:56:15,300 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-08 20:56:15,300 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680987375300"}]},"ts":"9223372036854775807"} 2023-04-08 20:56:15,307 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-08 20:56:15,309 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 20:56:15,310 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 77 msec 2023-04-08 20:56:15,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-08 20:56:15,336 INFO [Listener at localhost.localdomain/41823] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-08 20:56:15,350 INFO [Listener at localhost.localdomain/41823] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=228 (was 237), OpenFileDescriptor=550 (was 576), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=320 (was 235) - SystemLoadAverage LEAK? -, ProcessCount=166 (was 166), AvailableMemoryMB=3917 (was 3552) - AvailableMemoryMB LEAK? - 2023-04-08 20:56:15,361 INFO [Listener at localhost.localdomain/41823] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=228, OpenFileDescriptor=550, MaxFileDescriptor=60000, SystemLoadAverage=320, ProcessCount=166, AvailableMemoryMB=3916 2023-04-08 20:56:15,363 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-08 20:56:15,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-08 20:56:15,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-08 20:56:15,366 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-08 20:56:15,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-08 20:56:15,367 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-08 20:56:15,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-08 20:56:15,368 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028 2023-04-08 20:56:15,368 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028 empty. 2023-04-08 20:56:15,369 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028 2023-04-08 20:56:15,369 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-08 20:56:15,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-08 20:56:15,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-08 20:56:15,777 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-08 20:56:15,779 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => ebdeb8625c2b4ac57e00f30df2ca3028, NAME => 'TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp 2023-04-08 20:56:15,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-08 20:56:16,186 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 20:56:16,186 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing ebdeb8625c2b4ac57e00f30df2ca3028, disabling compactions & flushes 2023-04-08 20:56:16,186 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:16,186 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:16,186 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. after waiting 0 ms 2023-04-08 20:56:16,186 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:16,186 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:16,186 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:16,188 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-08 20:56:16,189 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987376189"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680987376189"}]},"ts":"1680987376189"} 2023-04-08 20:56:16,190 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-08 20:56:16,191 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-08 20:56:16,191 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680987376191"}]},"ts":"1680987376191"} 2023-04-08 20:56:16,192 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-08 20:56:16,193 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ebdeb8625c2b4ac57e00f30df2ca3028, ASSIGN}] 2023-04-08 20:56:16,195 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ebdeb8625c2b4ac57e00f30df2ca3028, ASSIGN 2023-04-08 20:56:16,196 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=ebdeb8625c2b4ac57e00f30df2ca3028, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,41091,1680987313601; forceNewPlan=false, retain=false 2023-04-08 20:56:16,347 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=ebdeb8625c2b4ac57e00f30df2ca3028, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:16,347 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987376347"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680987376347"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680987376347"}]},"ts":"1680987376347"} 2023-04-08 20:56:16,349 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601}] 2023-04-08 20:56:16,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-08 20:56:16,505 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:16,505 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ebdeb8625c2b4ac57e00f30df2ca3028, NAME => 'TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.', STARTKEY => '', ENDKEY => ''} 2023-04-08 20:56:16,505 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees ebdeb8625c2b4ac57e00f30df2ca3028 2023-04-08 20:56:16,505 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 20:56:16,505 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for ebdeb8625c2b4ac57e00f30df2ca3028 2023-04-08 20:56:16,505 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for ebdeb8625c2b4ac57e00f30df2ca3028 2023-04-08 20:56:16,507 INFO [StoreOpener-ebdeb8625c2b4ac57e00f30df2ca3028-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region ebdeb8625c2b4ac57e00f30df2ca3028 2023-04-08 20:56:16,509 DEBUG [StoreOpener-ebdeb8625c2b4ac57e00f30df2ca3028-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A 2023-04-08 20:56:16,509 DEBUG [StoreOpener-ebdeb8625c2b4ac57e00f30df2ca3028-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A 2023-04-08 20:56:16,510 INFO [StoreOpener-ebdeb8625c2b4ac57e00f30df2ca3028-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 20:56:16,510 INFO [StoreOpener-ebdeb8625c2b4ac57e00f30df2ca3028-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ebdeb8625c2b4ac57e00f30df2ca3028 columnFamilyName A 2023-04-08 20:56:16,511 INFO [StoreOpener-ebdeb8625c2b4ac57e00f30df2ca3028-1] regionserver.HStore(310): Store=ebdeb8625c2b4ac57e00f30df2ca3028/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 20:56:16,511 INFO [StoreOpener-ebdeb8625c2b4ac57e00f30df2ca3028-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region ebdeb8625c2b4ac57e00f30df2ca3028 2023-04-08 20:56:16,513 DEBUG [StoreOpener-ebdeb8625c2b4ac57e00f30df2ca3028-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B 2023-04-08 20:56:16,513 DEBUG [StoreOpener-ebdeb8625c2b4ac57e00f30df2ca3028-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B 2023-04-08 20:56:16,513 INFO [StoreOpener-ebdeb8625c2b4ac57e00f30df2ca3028-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 20:56:16,513 INFO [StoreOpener-ebdeb8625c2b4ac57e00f30df2ca3028-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ebdeb8625c2b4ac57e00f30df2ca3028 columnFamilyName B 2023-04-08 20:56:16,514 INFO [StoreOpener-ebdeb8625c2b4ac57e00f30df2ca3028-1] regionserver.HStore(310): Store=ebdeb8625c2b4ac57e00f30df2ca3028/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 20:56:16,514 INFO [StoreOpener-ebdeb8625c2b4ac57e00f30df2ca3028-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region ebdeb8625c2b4ac57e00f30df2ca3028 2023-04-08 20:56:16,516 DEBUG [StoreOpener-ebdeb8625c2b4ac57e00f30df2ca3028-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C 2023-04-08 20:56:16,516 DEBUG [StoreOpener-ebdeb8625c2b4ac57e00f30df2ca3028-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C 2023-04-08 20:56:16,516 INFO [StoreOpener-ebdeb8625c2b4ac57e00f30df2ca3028-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 20:56:16,516 INFO [StoreOpener-ebdeb8625c2b4ac57e00f30df2ca3028-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ebdeb8625c2b4ac57e00f30df2ca3028 columnFamilyName C 2023-04-08 20:56:16,518 INFO [StoreOpener-ebdeb8625c2b4ac57e00f30df2ca3028-1] regionserver.HStore(310): Store=ebdeb8625c2b4ac57e00f30df2ca3028/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 20:56:16,518 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:16,519 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028 2023-04-08 20:56:16,520 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028 2023-04-08 20:56:16,528 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-08 20:56:16,530 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for ebdeb8625c2b4ac57e00f30df2ca3028 2023-04-08 20:56:16,539 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-08 20:56:16,540 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened ebdeb8625c2b4ac57e00f30df2ca3028; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70953107, jitterRate=0.057283684611320496}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-08 20:56:16,540 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:16,541 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028., pid=30, masterSystemTime=1680987376501 2023-04-08 20:56:16,542 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:16,542 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:16,543 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=ebdeb8625c2b4ac57e00f30df2ca3028, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:16,543 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987376543"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680987376543"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680987376543"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680987376543"}]},"ts":"1680987376543"} 2023-04-08 20:56:16,545 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-08 20:56:16,546 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 in 195 msec 2023-04-08 20:56:16,547 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-08 20:56:16,547 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=ebdeb8625c2b4ac57e00f30df2ca3028, ASSIGN in 353 msec 2023-04-08 20:56:16,548 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-08 20:56:16,548 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680987376548"}]},"ts":"1680987376548"} 2023-04-08 20:56:16,549 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-08 20:56:16,551 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-08 20:56:16,552 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1870 sec 2023-04-08 20:56:17,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-08 20:56:17,472 INFO [Listener at localhost.localdomain/41823] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-08 20:56:17,473 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x73ef029c to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:56:17,478 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@f210b07, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:56:17,479 DEBUG [Listener at localhost.localdomain/41823] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:56:17,480 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44388, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:56:17,482 DEBUG [Listener at localhost.localdomain/41823] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-08 20:56:17,484 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55416, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-08 20:56:17,486 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x2135e03b to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:56:17,496 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2a45fd63, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:56:17,497 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x5f2b45f8 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:56:17,510 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5191421e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:56:17,510 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x24e1e878 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:56:17,514 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21e07d99, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:56:17,514 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x7e9921a7 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:56:17,518 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5590a424, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:56:17,518 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x5ffcf146 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:56:17,523 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@68df7707, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:56:17,523 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x07742ca6 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:56:17,530 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@14001400, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:56:17,531 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x353e11a8 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:56:17,544 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13091ab5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:56:17,544 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x12f583c6 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:56:17,552 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@57836361, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:56:17,553 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x6635f17a to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:56:17,564 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5860b5c9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:56:17,564 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x38204078 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:56:17,573 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@71d7d827, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:56:17,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:17,586 DEBUG [hconnection-0x2c2a1963-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:56:17,590 DEBUG [hconnection-0x76cab44c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:56:17,590 DEBUG [hconnection-0x24ea256f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:56:17,592 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44406, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:56:17,599 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44412, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:56:17,599 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44390, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:56:17,599 DEBUG [hconnection-0x1b34b5f7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:56:17,600 DEBUG [hconnection-0x704dc886-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:56:17,602 DEBUG [hconnection-0x7ea310a2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:56:17,602 DEBUG [hconnection-0x2aff60c3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:56:17,602 DEBUG [hconnection-0x11760b3b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:56:17,604 DEBUG [hconnection-0x486bf687-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:56:17,607 DEBUG [hconnection-0x1d4c649c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:56:17,607 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44430, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:56:17,609 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44444, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:56:17,610 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44464, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:56:17,612 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44448, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:56:17,616 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44428, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:56:17,616 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44446, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:56:17,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:17,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:17,616 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44472, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:56:17,618 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:17,618 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:17,618 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:17,618 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:17,618 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:17,619 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:17,620 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:17,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:17,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:17,620 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,621 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,621 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:17,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:17,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:17,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:17,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:17,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on ebdeb8625c2b4ac57e00f30df2ca3028 2023-04-08 20:56:17,634 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:17,634 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ebdeb8625c2b4ac57e00f30df2ca3028 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-08 20:56:17,635 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=A 2023-04-08 20:56:17,635 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:17,635 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=B 2023-04-08 20:56:17,635 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:17,635 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=C 2023-04-08 20:56:17,635 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:17,637 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:17,639 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:17,639 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:17,639 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:17,639 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:17,639 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:17,639 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:17,642 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:17,642 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:17,642 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:17,642 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:17,642 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:17,642 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:17,642 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,642 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:17,642 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:17,644 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,644 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:17,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:17,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:17,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:17,645 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:17,645 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:17,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:17,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:17,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:17,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:17,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ac5b91c[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:17,649 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:17,649 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:17,650 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/9711a2f388074f82a11d9d0a6665281c 2023-04-08 20:56:17,655 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:17,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:17,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:17,655 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:17,655 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:17,655 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,655 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:17,655 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:17,660 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680987377655, max timestamp=1680987377658Num uniques 859; 2023-04-08 20:56:17,660 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680987377655, max timestamp=1680987377658Num uniques 859; 2023-04-08 20:56:17,660 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 20:56:17,660 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 20:56:17,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680987377655, max timestamp=1680987377658Num uniques 859; 2023-04-08 20:56:17,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 20:56:17,662 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,662 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:17,662 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:17,662 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:17,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:17,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:17,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:17,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:17,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:17,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:17,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:17,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:17,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:17,666 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:17,666 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:17,666 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:17,666 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:17,666 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:17,671 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:17,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:17,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:17,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:17,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:17,672 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:17,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:17,674 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:17,674 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:17,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:17,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:17,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:17,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:17,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:17,674 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:17,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:17,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:17,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,680 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:17,680 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:17,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:17,680 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:17,680 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:17,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:17,680 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:17,680 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,680 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:17,680 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:17,680 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:17,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:17,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:17,686 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:17,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:17,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:17,686 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:17,686 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:17,699 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:17,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44430 deadline: 1680987437695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,700 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:17,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44448 deadline: 1680987437696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,700 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:17,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44428 deadline: 1680987437697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,700 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:17,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44472 deadline: 1680987437697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,700 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:17,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44412 deadline: 1680987437698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,787 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:17,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:17,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:17,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:17,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:17,790 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:17,790 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:17,790 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:17,790 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:17,790 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:17,791 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:17,791 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,791 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:17,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:17,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:17,792 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,792 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:17,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:17,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:17,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:17,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:17,793 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:17,793 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:17,793 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:17,793 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:17,793 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:17,793 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:17,793 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:17,793 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:17,794 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:17,794 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:17,794 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:17,794 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:17,794 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:17,794 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:17,794 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,794 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:17,794 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:17,795 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,795 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:17,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:17,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:17,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:17,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:17,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:17,797 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:17,797 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:17,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:17,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:17,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cd83819[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:17,797 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:17,797 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:17,798 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:17,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:17,799 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,799 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:17,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:17,799 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:17,799 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:17,799 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:17,800 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:17,800 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,800 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:17,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:17,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:17,800 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:17,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:17,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:17,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:17,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:17,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:17,802 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:17,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44430 deadline: 1680987437800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:17,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44448 deadline: 1680987437801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:17,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44412 deadline: 1680987437802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:17,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44472 deadline: 1680987437802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:17,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:17,804 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:17,804 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:17,804 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:17,804 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:17,804 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:17,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:17,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44428 deadline: 1680987437802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,805 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:17,805 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:17,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:17,805 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:17,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:17,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:17,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:17,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:17,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:17,806 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:17,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:17,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:17,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:17,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:17,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:17,807 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:17,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:17,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,808 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,808 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:17,809 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:17,809 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:17,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:17,809 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:17,809 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:17,809 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:17,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:17,809 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:17,809 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:17,809 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:17,809 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:17,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:17,809 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:17,809 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:17,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:17,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:17,909 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:17,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:17,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:17,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:17,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:17,912 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:17,913 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:17,913 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:17,913 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:17,913 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:17,913 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:17,914 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,914 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:17,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:17,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:17,914 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,914 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:17,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:17,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:17,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:17,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:17,920 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:17,921 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:17,921 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:17,921 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:17,921 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:17,921 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:17,921 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:17,921 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:17,921 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:17,921 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:17,921 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:17,921 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:17,921 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:17,921 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:17,921 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,921 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:17,921 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:17,931 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,931 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:17,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:17,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:17,932 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:17,932 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:17,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:17,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:17,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:17,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:17,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:17,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22db7b13[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:17,933 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:17,933 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:17,934 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:17,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:17,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:17,935 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:17,935 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:17,935 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,935 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:17,935 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:17,937 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:17,937 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,937 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:17,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:17,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:17,937 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:17,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:17,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:17,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:17,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:17,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:17,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:17,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:17,941 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:17,941 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:17,941 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:17,941 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:17,941 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:17,943 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:17,943 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:17,943 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:17,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:17,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:17,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:17,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:17,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:17,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:17,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:17,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:17,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:17,946 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:17,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:17,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:17,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,948 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:17,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:17,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,950 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,950 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:17,950 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:17,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:17,950 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:17,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:17,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:17,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:17,950 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:17,950 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:17,950 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:17,950 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:17,950 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:17,950 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:17,950 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:17,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:17,950 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:17,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:18,004 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:18,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44412 deadline: 1680987438004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,006 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:18,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44448 deadline: 1680987438004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,006 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:18,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44430 deadline: 1680987438004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,006 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:18,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44472 deadline: 1680987438005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,007 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:18,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44428 deadline: 1680987438006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,051 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:18,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:18,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:18,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:18,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:18,053 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:18,053 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:18,053 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:18,053 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:18,054 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:18,054 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:18,055 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,056 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:18,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:18,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:18,056 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,056 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:18,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:18,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:18,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:18,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:18,058 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:18,058 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:18,058 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:18,058 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:18,058 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:18,058 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:18,058 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:18,058 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:18,058 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:18,058 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:18,059 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:18,059 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:18,059 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:18,059 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:18,059 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,059 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:18,059 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:18,060 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:18,060 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:18,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:18,060 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:18,060 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:18,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:18,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:18,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:18,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:18,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:18,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4220db76[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:18,061 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:18,061 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:18,062 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:18,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:18,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:18,062 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:18,062 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:18,062 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:18,062 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,062 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:18,063 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,063 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:18,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:18,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:18,063 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:18,063 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:18,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:18,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:18,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:18,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:18,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:18,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:18,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:18,065 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:18,065 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:18,065 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:18,065 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:18,065 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:18,066 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:18,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:18,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:18,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:18,066 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:18,066 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:18,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:18,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:18,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:18,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:18,066 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:18,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:18,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:18,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:18,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:18,067 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:18,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:18,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,069 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,069 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:18,069 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:18,069 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:18,069 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,069 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:18,069 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:18,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:18,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:18,069 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:18,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:18,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:18,069 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:18,069 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:18,069 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:18,069 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:18,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:18,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:18,077 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/dd22993dfba649ac9987823e17a91ec5 2023-04-08 20:56:18,089 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/db9d492ec4994ce9b2ec51cd5a8e599c 2023-04-08 20:56:18,093 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/9711a2f388074f82a11d9d0a6665281c as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/9711a2f388074f82a11d9d0a6665281c 2023-04-08 20:56:18,098 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/9711a2f388074f82a11d9d0a6665281c, entries=150, sequenceid=11, filesize=11.6 K 2023-04-08 20:56:18,099 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/dd22993dfba649ac9987823e17a91ec5 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/dd22993dfba649ac9987823e17a91ec5 2023-04-08 20:56:18,106 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/dd22993dfba649ac9987823e17a91ec5, entries=150, sequenceid=11, filesize=11.6 K 2023-04-08 20:56:18,107 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/db9d492ec4994ce9b2ec51cd5a8e599c as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/db9d492ec4994ce9b2ec51cd5a8e599c 2023-04-08 20:56:18,113 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/db9d492ec4994ce9b2ec51cd5a8e599c, entries=150, sequenceid=11, filesize=11.6 K 2023-04-08 20:56:18,114 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for ebdeb8625c2b4ac57e00f30df2ca3028 in 480ms, sequenceid=11, compaction requested=false 2023-04-08 20:56:18,114 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-08 20:56:18,115 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:18,169 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:18,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:18,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:18,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:18,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:18,172 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:18,172 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:18,172 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:18,172 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:18,172 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:18,173 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:18,173 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:18,173 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:18,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:18,173 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,174 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:18,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:18,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:18,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:18,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:18,174 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:18,174 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:18,174 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:18,174 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:18,174 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:18,175 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:18,175 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:18,175 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:18,175 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:18,175 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:18,175 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ebdeb8625c2b4ac57e00f30df2ca3028 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-08 20:56:18,175 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=A 2023-04-08 20:56:18,175 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:56:18,175 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=B 2023-04-08 20:56:18,175 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:56:18,176 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=C 2023-04-08 20:56:18,176 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:56:18,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on ebdeb8625c2b4ac57e00f30df2ca3028 2023-04-08 20:56:18,307 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:18,318 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:18,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44430 deadline: 1680987438312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,318 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:18,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44428 deadline: 1680987438313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,319 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:18,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44472 deadline: 1680987438314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,322 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:18,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44448 deadline: 1680987438318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,322 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:18,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44412 deadline: 1680987438319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,421 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:18,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44430 deadline: 1680987438419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:18,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44428 deadline: 1680987438419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,422 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:18,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44472 deadline: 1680987438420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,425 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:18,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44412 deadline: 1680987438424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,425 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:18,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44448 deadline: 1680987438424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,582 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/14fc9486bc5a465a8923f86c5e75be88 2023-04-08 20:56:18,595 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/ecdbe5c948e640ddaf0ad076a0aa31a3 2023-04-08 20:56:18,623 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:18,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44430 deadline: 1680987438622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,623 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:18,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44428 deadline: 1680987438622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,625 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:18,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44472 deadline: 1680987438623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:18,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44448 deadline: 1680987438626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:18,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44412 deadline: 1680987438627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,927 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:18,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44428 deadline: 1680987438925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,929 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:18,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44430 deadline: 1680987438926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,931 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:18,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44472 deadline: 1680987438928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,932 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:18,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44412 deadline: 1680987438931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:18,933 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:18,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44448 deadline: 1680987438931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:19,014 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/fb70b25972cb44f3be382464c4bb7685 2023-04-08 20:56:19,019 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/14fc9486bc5a465a8923f86c5e75be88 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/14fc9486bc5a465a8923f86c5e75be88 2023-04-08 20:56:19,023 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/14fc9486bc5a465a8923f86c5e75be88, entries=150, sequenceid=48, filesize=11.6 K 2023-04-08 20:56:19,024 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/ecdbe5c948e640ddaf0ad076a0aa31a3 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/ecdbe5c948e640ddaf0ad076a0aa31a3 2023-04-08 20:56:19,030 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/ecdbe5c948e640ddaf0ad076a0aa31a3, entries=150, sequenceid=48, filesize=11.6 K 2023-04-08 20:56:19,032 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/fb70b25972cb44f3be382464c4bb7685 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/fb70b25972cb44f3be382464c4bb7685 2023-04-08 20:56:19,036 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/fb70b25972cb44f3be382464c4bb7685, entries=150, sequenceid=48, filesize=11.6 K 2023-04-08 20:56:19,037 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=53.67 KB/54960 for ebdeb8625c2b4ac57e00f30df2ca3028 in 861ms, sequenceid=48, compaction requested=false 2023-04-08 20:56:19,039 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:19,039 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:19,039 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:19,039 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:19,039 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:19,039 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:19,039 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:19,040 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:19,040 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:19,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:19,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:19,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:19,040 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:19,040 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:19,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:19,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:19,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:19,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:19,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:19,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:19,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@751a9777[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:19,043 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:19,043 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:19,044 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:19,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:19,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:19,045 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:19,045 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:19,045 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:19,045 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:19,045 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:19,046 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:19,046 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:19,046 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:19,046 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:19,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:19,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:19,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:19,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:19,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:19,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:19,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:19,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:19,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:19,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:19,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:19,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:19,049 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:19,049 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:19,049 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:19,049 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:19,049 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:19,050 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:19,050 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:19,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:19,051 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:19,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:19,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:19,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:19,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:19,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:19,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:19,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:19,051 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:19,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:19,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:19,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:19,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:19,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:19,054 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:19,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:19,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:19,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:19,063 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:19,063 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:19,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:19,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:19,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:19,063 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:19,064 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:19,064 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:19,064 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:19,064 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:19,064 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:19,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:19,063 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:19,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:19,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:19,064 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:19,064 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:19,064 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:19,164 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:19,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:19,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:19,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:19,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:19,170 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:19,170 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:19,170 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:19,170 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:19,171 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:19,171 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:19,171 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:19,171 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:19,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:19,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:19,172 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:19,172 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:19,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:19,172 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:19,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:19,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:19,173 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:19,183 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:19,183 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:19,184 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:19,184 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:19,184 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:19,184 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:19,184 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:19,184 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:19,184 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:19,184 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ebdeb8625c2b4ac57e00f30df2ca3028 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-08 20:56:19,184 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=A 2023-04-08 20:56:19,184 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:19,184 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=B 2023-04-08 20:56:19,184 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:19,184 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=C 2023-04-08 20:56:19,185 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:19,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on ebdeb8625c2b4ac57e00f30df2ca3028 2023-04-08 20:56:19,429 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:19,461 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:19,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44472 deadline: 1680987439458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:19,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680987379455, max timestamp=1680987379461Num uniques 859; 2023-04-08 20:56:19,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-08 20:56:19,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680987379455, max timestamp=1680987379461Num uniques 859; 2023-04-08 20:56:19,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-08 20:56:19,463 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680987379455, max timestamp=1680987379461Num uniques 859; 2023-04-08 20:56:19,463 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-08 20:56:19,499 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:19,499 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:19,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44448 deadline: 1680987439496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:19,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44430 deadline: 1680987439496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:19,499 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:19,499 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:19,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44428 deadline: 1680987439497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:19,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44412 deadline: 1680987439497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:19,567 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:19,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44472 deadline: 1680987439564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:19,601 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/316178808b204f6e92ac0b364666f9b4 2023-04-08 20:56:19,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:19,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44430 deadline: 1680987439601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:19,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:19,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44428 deadline: 1680987439601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:19,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:19,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44412 deadline: 1680987439601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:19,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:19,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44448 deadline: 1680987439601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:19,768 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:19,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44472 deadline: 1680987439768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:19,807 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:19,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44430 deadline: 1680987439804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:19,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:19,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44428 deadline: 1680987439805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:19,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:19,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44448 deadline: 1680987439805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:19,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:19,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44412 deadline: 1680987439806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,020 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/3d38a91e832e4038bc76664e38f02d8e 2023-04-08 20:56:20,059 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/91202d7d40ac45afbfe6344114aeec53 2023-04-08 20:56:20,068 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/316178808b204f6e92ac0b364666f9b4 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/316178808b204f6e92ac0b364666f9b4 2023-04-08 20:56:20,074 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:20,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44472 deadline: 1680987440072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,076 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/316178808b204f6e92ac0b364666f9b4, entries=150, sequenceid=59, filesize=11.6 K 2023-04-08 20:56:20,077 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/3d38a91e832e4038bc76664e38f02d8e as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/3d38a91e832e4038bc76664e38f02d8e 2023-04-08 20:56:20,083 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/3d38a91e832e4038bc76664e38f02d8e, entries=150, sequenceid=59, filesize=11.6 K 2023-04-08 20:56:20,084 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/91202d7d40ac45afbfe6344114aeec53 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/91202d7d40ac45afbfe6344114aeec53 2023-04-08 20:56:20,088 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/91202d7d40ac45afbfe6344114aeec53, entries=150, sequenceid=59, filesize=11.6 K 2023-04-08 20:56:20,089 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for ebdeb8625c2b4ac57e00f30df2ca3028 in 905ms, sequenceid=59, compaction requested=true 2023-04-08 20:56:20,089 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:20,089 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:20,089 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:20,089 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:20,089 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,089 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:20,089 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:20,090 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,090 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:20,091 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,091 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:20,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:20,094 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:20,094 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@788f613d[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:20,096 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,096 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:20,096 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:20,096 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:20,096 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,096 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:20,097 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:20,097 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:20,097 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:20,097 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:20,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:20,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:20,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:20,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:20,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:20,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:20,100 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:20,100 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:20,100 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:20,100 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:20,101 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:20,102 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,102 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,102 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:20,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:20,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:20,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:20,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:20,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:20,106 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:20,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,107 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:20,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,110 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:20,110 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:20,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:20,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:20,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:20,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:20,111 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,111 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:20,111 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:20,111 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,111 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:20,111 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,111 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:20,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:20,111 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,111 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:20,112 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:20,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:20,114 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ebdeb8625c2b4ac57e00f30df2ca3028 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-08 20:56:20,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on ebdeb8625c2b4ac57e00f30df2ca3028 2023-04-08 20:56:20,115 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=A 2023-04-08 20:56:20,115 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:56:20,115 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=B 2023-04-08 20:56:20,115 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:56:20,115 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=C 2023-04-08 20:56:20,115 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:56:20,126 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:20,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44412 deadline: 1680987440124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,127 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:20,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44448 deadline: 1680987440124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,128 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:20,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44428 deadline: 1680987440125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,129 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:20,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44430 deadline: 1680987440126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,212 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:20,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:20,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:20,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:20,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:20,215 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:20,215 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:20,215 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:20,215 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:20,215 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,215 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:20,216 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:20,216 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:20,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:20,216 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:20,216 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:20,216 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:20,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:20,216 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:20,217 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:20,217 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:20,218 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:20,218 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:20,218 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:20,219 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:20,219 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:20,219 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:20,219 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:20,219 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:20,219 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:20,219 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:20,219 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:20,219 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,219 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:20,219 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:20,220 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,220 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:20,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:20,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:20,221 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,221 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:20,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:20,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:20,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:20,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1758edc[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:20,222 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:20,222 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,223 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,223 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:20,223 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:20,223 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:20,223 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,223 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:20,224 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:20,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:20,224 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:20,224 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:20,224 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:20,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:20,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:20,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:20,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:20,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:20,226 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:20,226 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:20,226 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:20,226 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:20,226 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:20,227 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,227 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,227 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:20,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:20,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:20,228 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:20,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:20,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,229 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:20,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:20,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44412 deadline: 1680987440228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:20,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44448 deadline: 1680987440228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:20,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44428 deadline: 1680987440230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,231 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:20,231 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:20,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44430 deadline: 1680987440230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,231 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:20,231 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:20,232 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:20,232 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:20,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:20,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:20,232 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,232 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:20,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:20,232 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:20,232 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:20,232 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:20,232 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:20,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:20,332 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:20,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:20,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:20,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:20,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:20,335 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:20,335 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:20,335 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:20,335 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:20,336 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,336 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:20,337 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:20,337 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:20,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:20,337 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,337 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:20,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:20,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:20,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:20,338 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:20,338 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:20,338 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:20,338 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:20,338 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:20,338 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:20,339 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:20,339 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:20,339 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:20,339 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:20,339 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:20,339 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:20,339 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:20,339 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:20,339 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,339 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:20,339 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:20,340 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,340 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:20,340 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,340 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:20,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:20,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:20,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@454a0eef[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:20,341 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:20,341 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,342 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,342 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:20,342 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:20,342 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:20,342 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,342 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:20,343 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,343 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:20,343 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:20,343 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:20,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:20,345 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:20,345 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:20,345 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:20,345 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:20,345 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:20,346 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,346 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:20,346 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,349 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:20,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:20,350 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,355 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,355 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:20,355 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:20,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:20,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:20,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:20,355 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:20,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:20,355 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:20,355 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:20,355 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:20,355 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:20,355 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:20,355 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:20,355 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,355 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,433 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:20,434 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:20,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44448 deadline: 1680987440432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44412 deadline: 1680987440432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,434 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:20,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44428 deadline: 1680987440432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:20,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44430 deadline: 1680987440433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,455 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:20,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:20,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:20,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:20,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:20,458 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:20,458 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:20,458 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:20,458 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:20,459 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,459 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:20,459 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:20,459 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:20,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:20,460 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,460 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:20,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:20,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:20,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:20,460 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:20,460 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:20,460 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:20,460 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:20,461 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:20,461 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:20,461 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:20,461 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:20,461 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:20,461 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:20,461 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:20,461 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:20,461 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:20,461 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:20,461 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,461 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:20,461 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:20,462 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,462 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:20,462 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,462 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:20,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:20,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:20,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64ca881e[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:20,463 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:20,463 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,464 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,464 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:20,464 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,464 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:20,464 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:20,464 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:20,465 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,465 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:20,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:20,465 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:20,466 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:20,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:20,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:20,467 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:20,467 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:20,467 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:20,467 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:20,467 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:20,468 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,468 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:20,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:20,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,468 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:20,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:20,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:20,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:20,469 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:20,470 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:20,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,480 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:20,480 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:20,480 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:20,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:20,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:20,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:20,480 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:20,480 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:20,480 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:20,480 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:20,480 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:20,480 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:20,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:20,480 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,481 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,481 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,540 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/e11e43423d5840b6a82356f36ba9b352 2023-04-08 20:56:20,579 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:20,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44472 deadline: 1680987440576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,580 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:20,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:20,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:20,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:20,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:20,590 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:20,590 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:20,590 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:20,590 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:20,593 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,593 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:20,594 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,595 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,595 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:20,598 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:20,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:20,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:20,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:20,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:20,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:20,599 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:20,599 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:20,599 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:20,599 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:20,599 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:20,599 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:20,602 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:20,602 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:20,602 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:20,602 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:20,602 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:20,602 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:20,602 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:20,602 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:20,602 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,602 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:20,602 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:20,613 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,613 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:20,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:20,613 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,614 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:20,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:20,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:20,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:20,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:20,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@674f8b33[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:20,615 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:20,615 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,619 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,619 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:20,621 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-08 20:56:20,623 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,623 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:20,623 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:20,623 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:20,630 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,630 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:20,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:20,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:20,630 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:20,630 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:20,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:20,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:20,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:20,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:20,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:20,632 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:20,632 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:20,632 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:20,633 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:20,633 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:20,634 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:20,634 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,634 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:20,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:20,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:20,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:20,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:20,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:20,639 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:20,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,641 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,646 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:20,646 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:20,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:20,647 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:20,647 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:20,647 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:20,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:20,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:20,647 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:20,647 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,647 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,647 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,647 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:20,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:20,647 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:20,647 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:20,737 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:20,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44448 deadline: 1680987440735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,738 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:20,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44412 deadline: 1680987440736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,738 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:20,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44428 deadline: 1680987440737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,742 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:20,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44430 deadline: 1680987440738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,748 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:20,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:20,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:20,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:20,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:20,754 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:20,754 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:20,754 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:20,754 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:20,755 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,755 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:20,755 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:20,755 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:20,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:20,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:20,756 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,756 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:20,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:20,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:20,756 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:20,757 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:20,757 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:20,757 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:20,757 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:20,757 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:20,757 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:20,757 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:20,757 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:20,757 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:20,757 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:20,757 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:20,757 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:20,757 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:20,757 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,757 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:20,757 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:20,759 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,759 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:20,759 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,759 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:20,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:20,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:20,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a61ea68[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:20,760 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:20,760 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,761 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,761 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:20,761 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,761 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:20,761 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:20,761 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:20,762 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:20,762 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:20,762 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,762 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:20,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:20,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:20,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:20,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:20,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:20,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:20,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:20,763 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:20,763 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:20,763 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:20,763 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:20,764 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:20,764 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,764 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:20,764 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:20,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:20,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:20,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:20,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:20,765 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:20,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,766 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:20,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,768 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:20,768 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:20,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:20,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:20,769 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,769 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:20,769 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:20,769 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,769 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,769 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:20,769 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,769 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:20,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:20,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:20,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:20,769 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:20,770 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:20,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:20,870 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:20,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:20,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:20,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:20,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:20,873 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:20,873 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:20,873 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:20,873 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:20,874 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,874 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:20,874 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:20,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:20,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:20,874 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:20,875 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,875 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:20,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:20,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:20,875 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:20,878 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:20,878 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:20,878 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:20,878 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:20,878 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:20,878 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:20,878 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:20,878 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:20,878 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:20,879 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:20,879 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:20,879 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:20,879 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:20,879 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,879 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:20,879 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:20,888 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,888 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:20,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:20,888 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,888 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:20,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:20,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:20,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:20,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:20,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f49a8bd[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:20,890 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:20,890 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,893 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,893 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:20,893 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:20,893 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,893 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:20,893 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:20,894 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,894 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:20,894 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:20,894 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:20,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:20,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:20,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:20,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:20,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:20,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:20,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:20,897 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:20,898 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:20,898 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:20,898 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:20,898 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:20,899 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,899 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:20,899 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:20,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:20,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:20,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:20,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:20,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,899 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:20,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,901 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:20,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:20,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,903 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,903 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:20,903 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:20,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:20,903 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:20,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:20,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:20,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:20,903 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:20,904 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:20,904 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,904 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:20,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:20,904 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:20,903 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:20,904 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:20,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:20,904 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:20,960 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/9e7058a141394626bccfd63000b9c688 2023-04-08 20:56:21,004 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:21,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:21,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:21,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:21,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:21,010 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:21,010 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:21,010 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:21,010 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:21,010 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,011 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:21,011 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,011 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:21,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:21,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:21,012 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,012 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:21,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:21,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:21,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:21,012 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:21,017 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:21,017 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:21,018 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:21,018 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:21,018 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:21,021 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:21,021 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:21,021 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:21,021 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:21,021 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:21,021 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:21,022 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:21,022 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:21,022 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,022 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:21,022 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:21,030 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,030 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:21,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:21,030 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,030 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:21,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:21,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:21,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:21,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:21,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ff35132[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:21,032 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:21,032 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,034 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,035 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:21,035 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:21,035 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,035 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:21,035 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:21,036 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:21,036 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,036 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:21,036 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:21,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:21,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:21,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:21,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:21,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:21,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:21,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:21,039 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:21,039 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:21,039 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:21,039 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:21,039 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:21,040 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,040 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:21,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:21,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,040 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:21,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:21,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:21,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:21,042 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:21,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,046 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:21,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,055 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,055 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:21,055 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:21,055 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,055 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,055 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,055 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,055 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:21,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:21,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:21,056 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:21,056 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:21,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:21,056 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:21,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:21,056 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:21,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:21,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:21,156 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:21,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:21,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:21,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:21,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:21,160 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:21,160 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:21,160 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:21,160 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:21,160 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,160 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:21,161 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,161 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:21,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:21,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:21,161 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,161 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:21,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:21,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:21,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:21,162 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:21,171 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:21,172 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:21,172 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:21,172 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:21,172 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:21,174 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:21,174 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:21,174 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:21,174 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:21,174 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:21,174 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:21,174 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:21,174 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:21,174 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,174 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:21,174 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:21,176 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,176 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:21,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:21,176 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,176 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:21,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:21,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:21,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:21,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:21,177 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:21,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74fe59ec[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:21,178 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,178 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,178 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:21,178 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:21,178 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:21,178 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,178 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:21,180 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:21,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:21,180 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:21,180 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:21,180 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:21,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:21,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:21,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:21,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:21,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:21,184 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:21,184 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:21,184 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:21,184 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:21,184 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:21,185 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,185 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:21,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:21,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,185 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:21,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:21,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:21,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:21,186 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:21,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,189 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:21,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,196 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,196 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:21,196 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:21,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:21,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:21,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:21,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:21,197 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:21,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:21,197 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:21,197 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,197 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,197 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,197 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,197 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:21,197 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:21,197 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:21,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:21,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:21,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:21,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44448 deadline: 1680987441241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:21,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44428 deadline: 1680987441241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:21,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44412 deadline: 1680987441243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,247 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:21,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44430 deadline: 1680987441244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,297 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:21,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:21,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:21,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:21,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:21,302 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:21,302 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:21,302 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:21,302 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:21,302 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,303 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:21,303 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,303 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,303 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:21,304 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:21,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:21,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:21,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:21,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:21,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:21,305 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:21,306 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:21,306 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:21,306 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:21,306 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:21,306 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:21,310 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:21,310 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:21,310 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:21,310 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:21,310 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:21,310 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:21,310 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:21,310 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:21,310 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,310 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:21,310 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:21,318 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,318 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:21,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:21,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:21,319 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,319 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:21,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:21,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:21,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:21,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3189cfb1[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:21,321 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:21,321 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,323 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,323 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:21,323 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,323 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:21,323 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:21,323 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:21,325 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:21,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:21,325 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:21,325 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:21,325 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:21,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:21,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:21,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:21,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:21,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:21,327 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:21,327 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:21,327 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:21,327 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:21,327 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:21,328 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,328 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,328 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:21,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:21,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:21,329 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:21,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:21,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:21,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:21,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,331 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:21,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,335 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:21,335 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:21,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:21,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:21,336 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,336 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:21,336 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:21,336 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:21,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:21,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:21,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:21,336 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,337 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,337 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,337 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:21,337 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:21,337 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:21,395 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/f692e50ad2184121b1175edb4db4e42d 2023-04-08 20:56:21,399 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/e11e43423d5840b6a82356f36ba9b352 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/e11e43423d5840b6a82356f36ba9b352 2023-04-08 20:56:21,408 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/e11e43423d5840b6a82356f36ba9b352, entries=250, sequenceid=96, filesize=16.2 K 2023-04-08 20:56:21,409 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/9e7058a141394626bccfd63000b9c688 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/9e7058a141394626bccfd63000b9c688 2023-04-08 20:56:21,416 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/9e7058a141394626bccfd63000b9c688, entries=150, sequenceid=96, filesize=11.6 K 2023-04-08 20:56:21,416 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/f692e50ad2184121b1175edb4db4e42d as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/f692e50ad2184121b1175edb4db4e42d 2023-04-08 20:56:21,422 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/f692e50ad2184121b1175edb4db4e42d, entries=150, sequenceid=96, filesize=11.6 K 2023-04-08 20:56:21,423 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for ebdeb8625c2b4ac57e00f30df2ca3028 in 1309ms, sequenceid=96, compaction requested=true 2023-04-08 20:56:21,423 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:21,424 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:56:21,424 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:56:21,424 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 20:56:21,424 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:56:21,424 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 20:56:21,425 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 20:56:21,425 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52224 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 20:56:21,425 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): ebdeb8625c2b4ac57e00f30df2ca3028/B is initiating minor compaction (all files) 2023-04-08 20:56:21,425 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): ebdeb8625c2b4ac57e00f30df2ca3028/A is initiating minor compaction (all files) 2023-04-08 20:56:21,426 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ebdeb8625c2b4ac57e00f30df2ca3028/B in TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:21,426 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ebdeb8625c2b4ac57e00f30df2ca3028/A in TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:21,426 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/dd22993dfba649ac9987823e17a91ec5, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/ecdbe5c948e640ddaf0ad076a0aa31a3, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/3d38a91e832e4038bc76664e38f02d8e, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/9e7058a141394626bccfd63000b9c688] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp, totalSize=46.4 K 2023-04-08 20:56:21,426 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/9711a2f388074f82a11d9d0a6665281c, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/14fc9486bc5a465a8923f86c5e75be88, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/316178808b204f6e92ac0b364666f9b4, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/e11e43423d5840b6a82356f36ba9b352] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp, totalSize=51 K 2023-04-08 20:56:21,426 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting dd22993dfba649ac9987823e17a91ec5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1680987377618 2023-04-08 20:56:21,426 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 9711a2f388074f82a11d9d0a6665281c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1680987377618 2023-04-08 20:56:21,426 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting ecdbe5c948e640ddaf0ad076a0aa31a3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680987377680 2023-04-08 20:56:21,427 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 3d38a91e832e4038bc76664e38f02d8e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1680987378310 2023-04-08 20:56:21,427 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 14fc9486bc5a465a8923f86c5e75be88, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680987377680 2023-04-08 20:56:21,427 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 9e7058a141394626bccfd63000b9c688, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1680987379496 2023-04-08 20:56:21,428 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 316178808b204f6e92ac0b364666f9b4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1680987378310 2023-04-08 20:56:21,428 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting e11e43423d5840b6a82356f36ba9b352, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1680987379467 2023-04-08 20:56:21,437 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:21,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:21,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:21,439 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): ebdeb8625c2b4ac57e00f30df2ca3028#B#compaction#225 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:56:21,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:21,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:21,440 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:21,440 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:21,440 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:21,440 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:21,441 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,441 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:21,441 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:21,441 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:21,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:21,442 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,442 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:21,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:21,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,442 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): ebdeb8625c2b4ac57e00f30df2ca3028#A#compaction#226 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:56:21,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:21,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:21,442 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:21,443 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:21,443 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:21,443 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:21,443 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:21,443 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:21,443 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:21,443 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:21,443 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:21,443 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:21,444 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ebdeb8625c2b4ac57e00f30df2ca3028 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-08 20:56:21,444 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=A 2023-04-08 20:56:21,444 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:21,444 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=B 2023-04-08 20:56:21,444 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:21,444 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=C 2023-04-08 20:56:21,444 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:21,483 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/243009136427467c8c96a1655770570f 2023-04-08 20:56:21,526 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/58b971b7f8154748b32aa8568640d49e 2023-04-08 20:56:21,536 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/4969367a170d4f47bdcf9468b9cb3c67 2023-04-08 20:56:21,540 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/243009136427467c8c96a1655770570f as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/243009136427467c8c96a1655770570f 2023-04-08 20:56:21,544 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/243009136427467c8c96a1655770570f, entries=150, sequenceid=105, filesize=11.6 K 2023-04-08 20:56:21,544 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/58b971b7f8154748b32aa8568640d49e as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/58b971b7f8154748b32aa8568640d49e 2023-04-08 20:56:21,553 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/58b971b7f8154748b32aa8568640d49e, entries=150, sequenceid=105, filesize=11.6 K 2023-04-08 20:56:21,562 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/4969367a170d4f47bdcf9468b9cb3c67 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/4969367a170d4f47bdcf9468b9cb3c67 2023-04-08 20:56:21,579 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/4969367a170d4f47bdcf9468b9cb3c67, entries=150, sequenceid=105, filesize=11.6 K 2023-04-08 20:56:21,580 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=0 B/0 for ebdeb8625c2b4ac57e00f30df2ca3028 in 136ms, sequenceid=105, compaction requested=true 2023-04-08 20:56:21,580 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:21,580 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:21,580 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:21,580 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:21,581 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,582 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:21,582 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:21,583 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,583 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:21,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:21,583 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,583 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:21,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:21,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:21,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:21,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:21,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2397857d[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:21,585 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:21,585 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,586 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,586 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:21,590 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,590 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:21,590 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:21,590 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:21,591 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:21,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:21,591 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:21,591 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:21,591 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:21,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:21,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:21,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:21,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:21,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:21,594 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:21,594 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:21,594 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:21,594 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:21,594 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:21,596 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,596 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,596 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:21,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:21,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:21,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:21,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:21,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,597 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:21,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:21,599 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:21,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,601 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:21,601 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:21,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:21,601 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:21,601 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:21,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:21,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:21,601 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:21,601 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:21,601 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,601 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,601 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,601 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,601 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:21,601 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:21,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:21,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:21,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:21,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on ebdeb8625c2b4ac57e00f30df2ca3028 2023-04-08 20:56:21,608 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ebdeb8625c2b4ac57e00f30df2ca3028 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-08 20:56:21,608 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=A 2023-04-08 20:56:21,608 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:21,608 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=B 2023-04-08 20:56:21,608 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:21,608 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=C 2023-04-08 20:56:21,608 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:21,681 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680987381660, max timestamp=1680987381679Num uniques 859; 2023-04-08 20:56:21,681 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680987381660, max timestamp=1680987381679Num uniques 859; 2023-04-08 20:56:21,681 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680987381660, max timestamp=1680987381679Num uniques 859; 2023-04-08 20:56:21,681 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 20:56:21,681 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 20:56:21,681 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 20:56:21,701 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:21,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:21,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:21,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:21,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:21,705 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:21,705 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:21,705 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:21,705 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:21,706 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,706 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:21,706 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:21,706 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:21,707 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,707 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:21,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:21,708 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:21,708 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:21,708 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:21,708 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:21,708 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:21,708 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:21,710 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:21,710 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:21,710 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:21,710 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:21,710 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:21,710 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:21,710 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:21,710 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:21,710 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,710 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:21,710 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:21,712 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:21,712 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:21,713 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,713 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:21,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:21,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:21,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:21,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56183afc[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:21,715 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:21,715 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,716 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,716 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:21,716 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:21,716 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:21,717 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,717 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:21,717 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:21,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:21,717 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:21,717 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:21,717 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:21,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:21,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:21,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:21,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:21,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:21,720 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:21,720 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:21,720 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:21,720 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:21,720 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:21,722 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,722 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:21,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,722 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:21,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:21,723 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:21,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,726 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:21,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,730 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,730 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:21,730 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:21,730 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:21,730 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:21,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:21,730 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:21,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:21,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:21,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:21,730 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:21,730 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:21,730 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,730 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:21,731 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,731 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:21,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:21,765 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:21,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44472 deadline: 1680987441761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,831 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:21,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:21,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:21,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:21,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:21,834 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:21,834 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:21,834 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:21,834 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:21,835 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,835 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:21,835 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:21,835 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:21,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:21,835 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,836 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:21,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:21,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:21,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:21,836 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:21,836 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:21,836 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:21,836 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:21,836 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:21,836 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:21,838 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:21,838 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:21,838 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:21,838 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:21,838 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:21,838 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:21,838 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:21,838 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:21,838 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,838 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:21,838 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:21,839 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,839 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:21,839 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,839 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:21,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:21,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:21,840 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:21,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50af28ea[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:21,840 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,841 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,841 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:21,841 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:21,841 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,841 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:21,841 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:21,841 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,841 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:21,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:21,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:21,841 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:21,842 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:21,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:21,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:21,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:21,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:21,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:21,843 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:21,843 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:21,843 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:21,843 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:21,843 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:21,844 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,844 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:21,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,844 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:21,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:21,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:21,845 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:21,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:21,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,846 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,847 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,847 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:21,847 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:21,847 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:21,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:21,847 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:21,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:21,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:21,847 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:21,847 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,847 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,847 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,847 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,847 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:21,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:21,847 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:21,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:21,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:21,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:21,870 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:21,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44472 deadline: 1680987441870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,873 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/aa83af904119442f81ef55453a956583 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/aa83af904119442f81ef55453a956583 2023-04-08 20:56:21,880 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ebdeb8625c2b4ac57e00f30df2ca3028/B of ebdeb8625c2b4ac57e00f30df2ca3028 into aa83af904119442f81ef55453a956583(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:56:21,880 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:21,880 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028., storeName=ebdeb8625c2b4ac57e00f30df2ca3028/B, priority=12, startTime=1680987381424; duration=0sec 2023-04-08 20:56:21,881 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:56:21,881 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-08 20:56:21,882 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59430 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-08 20:56:21,882 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): ebdeb8625c2b4ac57e00f30df2ca3028/C is initiating minor compaction (all files) 2023-04-08 20:56:21,882 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ebdeb8625c2b4ac57e00f30df2ca3028/C in TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:21,882 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/db9d492ec4994ce9b2ec51cd5a8e599c, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/fb70b25972cb44f3be382464c4bb7685, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/91202d7d40ac45afbfe6344114aeec53, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/f692e50ad2184121b1175edb4db4e42d, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/4969367a170d4f47bdcf9468b9cb3c67] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp, totalSize=58.0 K 2023-04-08 20:56:21,882 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting db9d492ec4994ce9b2ec51cd5a8e599c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1680987377618 2023-04-08 20:56:21,883 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting fb70b25972cb44f3be382464c4bb7685, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680987377680 2023-04-08 20:56:21,883 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 91202d7d40ac45afbfe6344114aeec53, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1680987378310 2023-04-08 20:56:21,883 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting f692e50ad2184121b1175edb4db4e42d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1680987379496 2023-04-08 20:56:21,884 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 4969367a170d4f47bdcf9468b9cb3c67, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1680987380122 2023-04-08 20:56:21,884 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/320490a1920e47d4ba982f867f78692e as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/320490a1920e47d4ba982f867f78692e 2023-04-08 20:56:21,896 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ebdeb8625c2b4ac57e00f30df2ca3028/A of ebdeb8625c2b4ac57e00f30df2ca3028 into 320490a1920e47d4ba982f867f78692e(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:56:21,896 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:21,896 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028., storeName=ebdeb8625c2b4ac57e00f30df2ca3028/A, priority=12, startTime=1680987381424; duration=0sec 2023-04-08 20:56:21,896 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:56:21,898 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): ebdeb8625c2b4ac57e00f30df2ca3028#C#compaction#231 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:56:21,945 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/667d8062e47d4252b66948dfbd2fc0da as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/667d8062e47d4252b66948dfbd2fc0da 2023-04-08 20:56:21,948 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:21,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:21,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:21,953 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ebdeb8625c2b4ac57e00f30df2ca3028/C of ebdeb8625c2b4ac57e00f30df2ca3028 into 667d8062e47d4252b66948dfbd2fc0da(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:56:21,953 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:21,953 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028., storeName=ebdeb8625c2b4ac57e00f30df2ca3028/C, priority=11, startTime=1680987381424; duration=0sec 2023-04-08 20:56:21,953 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:56:21,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:21,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:21,958 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:21,958 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:21,958 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:21,959 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:21,959 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,959 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:21,961 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,961 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:21,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:21,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:21,962 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:21,962 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:21,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:21,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:21,963 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:21,964 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:21,964 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:21,964 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:21,964 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:21,964 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:21,964 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:21,964 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:21,964 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:21,964 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:21,965 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:21,965 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:21,965 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:21,965 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:21,965 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,965 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:21,965 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:21,968 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,972 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:21,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:21,973 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,973 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:21,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:21,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:21,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:21,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:21,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f249109[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:21,982 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:21,982 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,984 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:21,984 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:21,984 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:21,984 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:21,985 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:21,985 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:21,985 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:21,985 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,988 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:21,988 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:21,988 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:21,988 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:21,988 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:21,988 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:21,988 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:21,989 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:21,989 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,989 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:21,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:21,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:21,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:21,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:21,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:21,991 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:21,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:21,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:21,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,000 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:22,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:22,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,010 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,010 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:22,010 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:22,010 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:22,010 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:22,010 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:22,010 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,010 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:22,010 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:22,010 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:22,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:22,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:22,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:22,014 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:22,014 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:22,037 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/b442862c6ac84445b7f7063885d859b4 2023-04-08 20:56:22,077 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:22,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44472 deadline: 1680987442073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,115 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:22,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:22,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:22,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:22,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:22,118 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:22,118 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:22,118 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:22,118 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:22,119 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,119 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:22,121 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,121 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:22,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:22,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:22,122 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,122 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:22,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:22,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:22,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:22,122 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:22,126 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:22,126 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:22,126 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:22,126 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:22,126 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:22,127 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:22,127 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:22,127 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:22,127 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:22,127 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:22,127 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:22,127 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:22,127 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:22,127 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,127 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:22,127 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:22,135 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,135 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:22,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:22,135 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,135 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:22,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:22,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:22,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:22,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:22,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@300745fd[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:22,137 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:22,137 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,143 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,143 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:22,143 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:22,143 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:22,145 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,145 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:22,145 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,145 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:22,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:22,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:22,146 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:22,146 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:22,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:22,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:22,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:22,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:22,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:22,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:22,152 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:22,153 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:22,153 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:22,153 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:22,153 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:22,160 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:22,160 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,160 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:22,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:22,161 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:22,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:22,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:22,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:22,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:22,164 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:22,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:22,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:22,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,169 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:22,169 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:22,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:22,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:22,170 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,170 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:22,170 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:22,170 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,170 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,170 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,170 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,170 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:22,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:22,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:22,170 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:22,170 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:22,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:22,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:22,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:22,248 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:22,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44412 deadline: 1680987442246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,254 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:22,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44428 deadline: 1680987442251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,254 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:22,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44448 deadline: 1680987442252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,259 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:22,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44430 deadline: 1680987442257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,271 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:22,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:22,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:22,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:22,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:22,284 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:22,284 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:22,284 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:22,284 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:22,285 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,285 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:22,286 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,286 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:22,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:22,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:22,286 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,286 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:22,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:22,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:22,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:22,289 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:22,298 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:22,298 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:22,298 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:22,298 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:22,298 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:22,306 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:22,306 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:22,307 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:22,307 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:22,307 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:22,307 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:22,307 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:22,307 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:22,307 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,307 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:22,307 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:22,308 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,308 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:22,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:22,308 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,308 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:22,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:22,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:22,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:22,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:22,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:22,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27df46db[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:22,310 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:22,310 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,311 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,311 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:22,311 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:22,311 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:22,311 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,311 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:22,312 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:22,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:22,312 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:22,312 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:22,312 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:22,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:22,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:22,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:22,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:22,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:22,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:22,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:22,318 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:22,318 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:22,318 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:22,318 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:22,318 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:22,323 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,324 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:22,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,324 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:22,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:22,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:22,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:22,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:22,328 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:22,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:22,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:22,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,335 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:22,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:22,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,338 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:22,338 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:22,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:22,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:22,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:22,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:22,340 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,340 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:22,340 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:22,340 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,340 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,340 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,340 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,340 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:22,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:22,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:22,340 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:22,340 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:22,383 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:22,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44472 deadline: 1680987442381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,440 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:22,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:22,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:22,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:22,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:22,446 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:22,446 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:22,446 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:22,446 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:22,447 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,447 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:22,448 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:22,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:22,448 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:22,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:22,449 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,449 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:22,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:22,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:22,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:22,464 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:22,464 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:22,464 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:22,464 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:22,464 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:22,466 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:22,466 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:22,466 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:22,466 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:22,466 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:22,466 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:22,466 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:22,466 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:22,466 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,466 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:22,466 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:22,469 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,469 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:22,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:22,470 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,470 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:22,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:22,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:22,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:22,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:22,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:22,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e1ec2b2[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:22,474 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:22,474 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,480 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,480 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:22,480 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:22,480 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:22,481 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,481 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:22,484 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/5e30a0ab3e4f4a05986f819531c9e577 2023-04-08 20:56:22,485 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:22,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:22,488 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:22,488 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:22,488 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:22,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:22,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:22,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:22,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:22,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:22,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:22,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:22,500 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:22,500 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:22,500 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:22,500 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:22,500 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:22,510 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,510 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:22,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:22,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,510 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:22,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:22,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:22,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:22,512 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:22,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:22,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:22,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,513 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:22,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:22,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,522 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:22,522 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,522 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:22,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:22,522 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:22,522 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:22,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:22,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:22,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:22,522 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:22,522 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,522 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,522 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,522 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:22,522 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,522 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:22,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:22,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:22,623 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:22,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:22,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:22,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:22,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:22,630 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:22,630 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:22,630 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:22,630 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:22,630 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,630 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:22,632 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:22,632 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:22,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:22,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:22,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,633 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,633 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:22,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:22,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:22,634 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:22,636 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:22,636 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:22,636 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:22,637 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:22,637 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:22,639 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:22,639 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:22,647 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:22,647 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:22,647 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:22,647 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:22,647 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:22,647 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:22,647 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,647 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:22,647 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:22,649 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,649 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:22,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:22,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:22,649 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,649 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:22,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:22,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:22,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:22,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:22,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b9a6a1[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:22,653 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:22,653 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,655 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,655 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:22,655 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:22,655 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:22,655 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,655 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:22,656 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:22,656 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:22,656 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:22,657 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:22,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:22,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:22,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:22,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:22,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:22,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:22,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:22,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:22,663 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:22,663 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:22,663 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:22,663 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:22,664 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:22,668 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,668 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,668 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:22,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:22,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:22,668 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:22,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:22,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:22,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:22,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:22,670 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:22,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:22,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:22,678 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,678 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:22,678 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:22,678 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,678 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,678 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,678 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:22,678 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:22,678 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:22,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:22,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:22,678 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:22,678 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:22,678 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:22,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:22,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:22,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:22,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:22,779 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:22,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:22,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:22,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:22,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:22,784 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:22,784 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:22,784 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:22,784 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:22,784 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,784 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:22,785 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:22,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:22,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:22,786 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:22,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,787 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,787 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:22,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:22,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:22,787 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:22,794 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:22,794 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:22,794 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:22,794 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:22,794 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:22,803 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:22,803 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:22,803 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:22,803 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:22,803 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:22,803 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:22,803 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:22,803 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:22,803 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,803 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:22,803 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:22,805 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,805 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:22,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:22,806 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,806 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:22,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:22,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:22,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:22,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:22,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:22,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f1b3346[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:22,807 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:22,807 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,809 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,809 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:22,809 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:22,809 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:22,811 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,811 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:22,812 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:22,812 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:22,812 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:22,812 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:22,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:22,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:22,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:22,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:22,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:22,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:22,819 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:22,819 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:22,819 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:22,819 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:22,819 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:22,826 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,826 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:22,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:22,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,826 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:22,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:22,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:22,827 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:22,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:22,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:22,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:22,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,829 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:22,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:22,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,835 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,835 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:22,835 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:22,835 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:22,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:22,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:22,835 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:22,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:22,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:22,835 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:22,835 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:22,835 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:22,835 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,835 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,835 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,835 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:22,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:22,889 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:22,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44472 deadline: 1680987442888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,935 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:22,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:22,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:22,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:22,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:22,938 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:22,938 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:22,938 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:22,938 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:22,939 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,939 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:22,939 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:22,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:22,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:22,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:22,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:22,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:22,940 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,940 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:22,940 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:22,941 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:22,941 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:22,941 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:22,941 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:22,941 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,941 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:22,941 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:22,941 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:22,941 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:22,941 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:22,941 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:22,941 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:22,941 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:22,941 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:22,941 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,941 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:22,941 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:22,942 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,942 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:22,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:22,942 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,942 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:22,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:22,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:22,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:22,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:22,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:22,943 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:22,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52af45e0[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:22,944 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,944 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,944 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:22,944 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:22,944 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:22,944 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,944 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:22,946 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,946 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:22,946 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:22,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:22,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:22,946 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:22,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:22,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:22,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:22,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:22,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:22,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:22,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:22,950 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:22,950 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:22,950 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:22,950 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:22,950 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:22,951 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,951 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:22,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,951 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:22,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:22,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:22,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:22,952 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:22,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:22,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:22,954 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:22,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:22,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:22,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,956 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:22,956 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:22,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:22,956 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:22,956 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,956 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:22,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:22,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:22,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:22,956 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:22,956 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/314678b7962340c3ba02e10fd0f7ed5b 2023-04-08 20:56:22,957 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:22,957 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:22,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:22,957 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,957 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:22,957 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:22,957 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:22,963 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/b442862c6ac84445b7f7063885d859b4 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/b442862c6ac84445b7f7063885d859b4 2023-04-08 20:56:22,966 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/b442862c6ac84445b7f7063885d859b4, entries=150, sequenceid=115, filesize=11.6 K 2023-04-08 20:56:22,967 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/5e30a0ab3e4f4a05986f819531c9e577 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/5e30a0ab3e4f4a05986f819531c9e577 2023-04-08 20:56:22,970 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/5e30a0ab3e4f4a05986f819531c9e577, entries=150, sequenceid=115, filesize=11.6 K 2023-04-08 20:56:22,970 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/314678b7962340c3ba02e10fd0f7ed5b as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/314678b7962340c3ba02e10fd0f7ed5b 2023-04-08 20:56:22,973 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/314678b7962340c3ba02e10fd0f7ed5b, entries=150, sequenceid=115, filesize=11.6 K 2023-04-08 20:56:22,974 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for ebdeb8625c2b4ac57e00f30df2ca3028 in 1366ms, sequenceid=115, compaction requested=true 2023-04-08 20:56:22,974 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:22,974 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:56:22,974 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:56:22,974 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:56:22,974 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:56:22,975 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:56:22,975 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:56:22,975 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:56:22,975 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): ebdeb8625c2b4ac57e00f30df2ca3028/A is initiating minor compaction (all files) 2023-04-08 20:56:22,975 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): ebdeb8625c2b4ac57e00f30df2ca3028/B is initiating minor compaction (all files) 2023-04-08 20:56:22,976 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ebdeb8625c2b4ac57e00f30df2ca3028/A in TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:22,976 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ebdeb8625c2b4ac57e00f30df2ca3028/B in TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:22,976 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/320490a1920e47d4ba982f867f78692e, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/243009136427467c8c96a1655770570f, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/b442862c6ac84445b7f7063885d859b4] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp, totalSize=35.0 K 2023-04-08 20:56:22,976 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/aa83af904119442f81ef55453a956583, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/58b971b7f8154748b32aa8568640d49e, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/5e30a0ab3e4f4a05986f819531c9e577] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp, totalSize=35.0 K 2023-04-08 20:56:22,977 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting aa83af904119442f81ef55453a956583, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1680987379496 2023-04-08 20:56:22,977 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 320490a1920e47d4ba982f867f78692e, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1680987379496 2023-04-08 20:56:22,977 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 58b971b7f8154748b32aa8568640d49e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1680987380122 2023-04-08 20:56:22,977 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 243009136427467c8c96a1655770570f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1680987380122 2023-04-08 20:56:22,978 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 5e30a0ab3e4f4a05986f819531c9e577, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1680987381600 2023-04-08 20:56:22,978 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting b442862c6ac84445b7f7063885d859b4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1680987381600 2023-04-08 20:56:22,983 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): ebdeb8625c2b4ac57e00f30df2ca3028#B#compaction#234 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:56:22,986 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): ebdeb8625c2b4ac57e00f30df2ca3028#A#compaction#235 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:56:23,007 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/22bba28af2d74afc8929b38072562408 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/22bba28af2d74afc8929b38072562408 2023-04-08 20:56:23,010 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ebdeb8625c2b4ac57e00f30df2ca3028/B of ebdeb8625c2b4ac57e00f30df2ca3028 into 22bba28af2d74afc8929b38072562408(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:56:23,010 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:23,010 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028., storeName=ebdeb8625c2b4ac57e00f30df2ca3028/B, priority=13, startTime=1680987382974; duration=0sec 2023-04-08 20:56:23,011 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:56:23,011 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-08 20:56:23,011 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-08 20:56:23,011 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-08 20:56:23,011 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. because compaction request was cancelled 2023-04-08 20:56:23,057 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:23,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:23,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:23,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:23,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:23,062 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:23,062 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:23,062 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:23,062 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:23,062 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:23,062 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:23,063 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:23,063 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:23,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:23,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:23,063 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:23,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:23,064 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:23,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:23,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:23,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:23,064 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:23,064 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:23,064 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:23,065 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:23,065 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:23,065 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:23,065 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:23,065 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:23,065 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:23,065 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:23,065 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ebdeb8625c2b4ac57e00f30df2ca3028 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-08 20:56:23,066 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=A 2023-04-08 20:56:23,066 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:56:23,066 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=B 2023-04-08 20:56:23,066 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:56:23,066 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=C 2023-04-08 20:56:23,066 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:56:23,100 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=156 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/de211dcb0b264fa29a320d6f603b64f2 2023-04-08 20:56:23,404 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/bbf9337be92148f4bcf0dc820d7390c8 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/bbf9337be92148f4bcf0dc820d7390c8 2023-04-08 20:56:23,409 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ebdeb8625c2b4ac57e00f30df2ca3028/A of ebdeb8625c2b4ac57e00f30df2ca3028 into bbf9337be92148f4bcf0dc820d7390c8(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:56:23,409 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:23,409 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028., storeName=ebdeb8625c2b4ac57e00f30df2ca3028/A, priority=13, startTime=1680987382974; duration=0sec 2023-04-08 20:56:23,409 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:56:23,512 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=156 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/3fe398aa3624424181e0724599ec72fb 2023-04-08 20:56:23,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on ebdeb8625c2b4ac57e00f30df2ca3028 2023-04-08 20:56:23,892 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:23,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:23,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44472 deadline: 1680987443908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:23,921 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=156 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/f32dfaac5bfe4f0a8e2d669c264e6f84 2023-04-08 20:56:23,924 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/de211dcb0b264fa29a320d6f603b64f2 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/de211dcb0b264fa29a320d6f603b64f2 2023-04-08 20:56:23,928 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/de211dcb0b264fa29a320d6f603b64f2, entries=150, sequenceid=156, filesize=11.8 K 2023-04-08 20:56:23,929 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/3fe398aa3624424181e0724599ec72fb as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/3fe398aa3624424181e0724599ec72fb 2023-04-08 20:56:23,933 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/3fe398aa3624424181e0724599ec72fb, entries=150, sequenceid=156, filesize=11.8 K 2023-04-08 20:56:23,934 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/f32dfaac5bfe4f0a8e2d669c264e6f84 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/f32dfaac5bfe4f0a8e2d669c264e6f84 2023-04-08 20:56:23,938 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/f32dfaac5bfe4f0a8e2d669c264e6f84, entries=150, sequenceid=156, filesize=11.8 K 2023-04-08 20:56:23,938 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for ebdeb8625c2b4ac57e00f30df2ca3028 in 873ms, sequenceid=156, compaction requested=true 2023-04-08 20:56:23,939 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:23,939 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:23,939 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:23,939 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:23,939 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:23,939 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:23,939 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:23,940 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:23,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:23,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:23,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:23,940 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:23,940 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:23,940 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:23,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:23,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:23,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:23,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:23,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:23,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:23,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2adc1655[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:23,941 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:23,941 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:23,942 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:23,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:23,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:23,942 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:23,942 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:23,942 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:23,942 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:23,942 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:23,943 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:23,943 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:23,943 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:23,943 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:23,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:23,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:23,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:23,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:23,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:23,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:23,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:23,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:23,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:23,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:23,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:23,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:23,945 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:23,945 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:23,945 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:23,945 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:23,945 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:23,946 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:23,946 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:23,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:23,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:23,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:23,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:23,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:23,946 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:23,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:23,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:23,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:23,946 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:23,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:23,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:23,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:23,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:23,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:23,947 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:23,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:23,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:23,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:23,949 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:23,949 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:23,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:23,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:23,949 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:23,949 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:23,949 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:23,949 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:23,949 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:23,949 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:23,949 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:23,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:23,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:23,949 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:23,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:23,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:23,950 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:23,950 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:24,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on ebdeb8625c2b4ac57e00f30df2ca3028 2023-04-08 20:56:24,013 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ebdeb8625c2b4ac57e00f30df2ca3028 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-08 20:56:24,014 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=A 2023-04-08 20:56:24,014 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:24,014 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=B 2023-04-08 20:56:24,014 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:24,014 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=C 2023-04-08 20:56:24,014 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:24,020 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/da48d5fd36794e5e9feb1ca4c3fffb44 2023-04-08 20:56:24,050 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:24,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:24,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:24,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:24,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:24,052 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:24,052 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:24,052 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:24,052 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:24,053 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,053 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:24,053 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,054 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:24,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:24,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:24,054 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,054 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:24,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:24,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:24,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:24,054 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:24,055 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:24,055 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:24,055 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:24,055 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:24,055 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:24,055 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:24,055 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:24,055 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:24,055 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:24,055 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:24,055 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:24,055 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:24,055 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:24,055 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,055 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:24,055 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:24,056 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,056 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:24,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:24,056 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,056 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:24,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:24,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:24,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:24,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:24,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:24,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62183101[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:24,062 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:24,062 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,072 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,072 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:24,072 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,073 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:24,073 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:24,073 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:24,076 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:24,076 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:24,076 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:24,077 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,077 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:24,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44472 deadline: 1680987444073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:24,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:24,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680987384057, max timestamp=1680987384071Num uniques 859; 2023-04-08 20:56:24,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-08 20:56:24,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:24,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680987384057, max timestamp=1680987384071Num uniques 859; 2023-04-08 20:56:24,079 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-08 20:56:24,079 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680987384057, max timestamp=1680987384071Num uniques 859; 2023-04-08 20:56:24,079 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-08 20:56:24,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:24,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:24,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:24,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:24,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:24,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:24,082 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:24,082 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:24,082 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:24,082 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:24,082 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:24,088 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,088 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:24,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:24,088 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:24,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:24,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:24,089 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:24,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:24,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:24,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:24,092 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:24,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:24,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,101 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:24,101 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:24,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:24,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:24,102 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,102 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:24,102 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:24,102 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,102 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,102 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:24,102 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:24,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:24,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:24,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:24,102 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:24,106 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:24,106 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:24,207 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:24,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:24,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:24,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:24,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:24,222 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:24,222 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:24,222 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:24,222 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:24,223 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,223 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:24,224 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:24,224 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:24,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:24,227 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,227 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:24,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:24,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:24,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:24,228 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:24,237 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:24,237 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:24,237 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:24,237 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:24,237 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:24,247 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:24,247 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:24,247 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:24,247 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:24,247 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:24,248 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:24,248 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:24,248 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:24,248 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,248 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:24,248 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:24,249 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,249 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:24,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:24,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:24,249 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,249 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:24,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:24,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:24,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:24,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:24,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@344fbf07[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:24,252 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:24,252 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,253 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,253 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:24,253 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:24,253 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:24,254 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,255 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:24,255 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:24,255 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:24,255 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:24,255 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:24,255 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:24,255 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:24,255 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:24,255 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:24,255 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:24,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:24,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44412 deadline: 1680987444253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,257 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,257 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:24,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:24,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,257 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:24,262 DEBUG [Thread-1669] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4134 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028., hostname=jenkins-hbase20.apache.org,41091,1680987313601, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 20:56:24,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:24,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:24,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:24,263 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:24,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:24,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:24,264 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:24,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:24,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,267 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:24,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987444262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:24,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44428 deadline: 1680987444269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,275 DEBUG [Thread-1665] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028., hostname=jenkins-hbase20.apache.org,41091,1680987313601, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 20:56:24,282 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,282 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:24,282 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:24,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:24,282 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:24,282 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,282 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,282 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:24,283 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:24,283 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,283 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:24,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:24,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:24,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:24,284 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:24,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44448 deadline: 1680987444283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:24,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:24,284 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:24,284 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:24,286 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:24,286 DEBUG [Thread-1661] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4160 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028., hostname=jenkins-hbase20.apache.org,41091,1680987313601, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 20:56:24,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44430 deadline: 1680987444283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,287 DEBUG [Thread-1663] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4160 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028., hostname=jenkins-hbase20.apache.org,41091,1680987313601, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 20:56:24,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:24,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987444368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,385 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:24,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:24,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:24,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:24,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:24,406 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:24,406 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:24,406 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:24,406 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:24,407 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,407 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:24,408 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,408 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:24,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:24,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:24,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:24,409 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,409 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:24,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:24,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:24,410 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:24,413 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:24,413 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:24,413 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:24,413 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:24,413 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:24,413 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:24,413 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:24,413 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:24,413 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:24,414 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:24,414 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:24,414 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:24,414 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:24,414 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,414 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:24,414 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:24,415 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,415 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:24,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:24,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:24,416 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,416 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:24,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:24,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:24,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:24,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:24,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ce76c2b[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:24,421 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:24,421 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,426 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,426 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:24,427 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:24,427 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:24,428 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/8fd130d9f8ee4bac98c987b38e0b4577 2023-04-08 20:56:24,430 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,430 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:24,430 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:24,430 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:24,431 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:24,431 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:24,431 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:24,431 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:24,431 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:24,431 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:24,431 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:24,432 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,432 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:24,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:24,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:24,433 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:24,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:24,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:24,434 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:24,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:24,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:24,435 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:24,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,437 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:24,437 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:24,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:24,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:24,437 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,437 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:24,437 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:24,437 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:24,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:24,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:24,437 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,438 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:24,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:24,438 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:24,438 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:24,438 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,438 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:24,455 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/7fa87866a6654accb0cf57a730b16e41 2023-04-08 20:56:24,461 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/da48d5fd36794e5e9feb1ca4c3fffb44 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/da48d5fd36794e5e9feb1ca4c3fffb44 2023-04-08 20:56:24,465 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/da48d5fd36794e5e9feb1ca4c3fffb44, entries=150, sequenceid=168, filesize=11.8 K 2023-04-08 20:56:24,467 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/8fd130d9f8ee4bac98c987b38e0b4577 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/8fd130d9f8ee4bac98c987b38e0b4577 2023-04-08 20:56:24,471 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/8fd130d9f8ee4bac98c987b38e0b4577, entries=150, sequenceid=168, filesize=11.8 K 2023-04-08 20:56:24,472 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/7fa87866a6654accb0cf57a730b16e41 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/7fa87866a6654accb0cf57a730b16e41 2023-04-08 20:56:24,477 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/7fa87866a6654accb0cf57a730b16e41, entries=150, sequenceid=168, filesize=11.8 K 2023-04-08 20:56:24,478 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=126.28 KB/129315 for ebdeb8625c2b4ac57e00f30df2ca3028 in 465ms, sequenceid=168, compaction requested=true 2023-04-08 20:56:24,478 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:24,478 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:56:24,478 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:56:24,479 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:56:24,479 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:56:24,479 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:56:24,479 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36198 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:56:24,479 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): ebdeb8625c2b4ac57e00f30df2ca3028/A is initiating minor compaction (all files) 2023-04-08 20:56:24,479 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ebdeb8625c2b4ac57e00f30df2ca3028/A in TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:24,480 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/bbf9337be92148f4bcf0dc820d7390c8, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/de211dcb0b264fa29a320d6f603b64f2, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/da48d5fd36794e5e9feb1ca4c3fffb44] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp, totalSize=35.3 K 2023-04-08 20:56:24,480 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36198 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:56:24,480 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): ebdeb8625c2b4ac57e00f30df2ca3028/B is initiating minor compaction (all files) 2023-04-08 20:56:24,480 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ebdeb8625c2b4ac57e00f30df2ca3028/B in TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:24,480 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting bbf9337be92148f4bcf0dc820d7390c8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1680987381600 2023-04-08 20:56:24,480 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/22bba28af2d74afc8929b38072562408, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/3fe398aa3624424181e0724599ec72fb, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/8fd130d9f8ee4bac98c987b38e0b4577] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp, totalSize=35.3 K 2023-04-08 20:56:24,480 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting de211dcb0b264fa29a320d6f603b64f2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1680987381716 2023-04-08 20:56:24,480 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 22bba28af2d74afc8929b38072562408, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1680987381600 2023-04-08 20:56:24,480 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting da48d5fd36794e5e9feb1ca4c3fffb44, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1680987383903 2023-04-08 20:56:24,481 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 3fe398aa3624424181e0724599ec72fb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1680987381716 2023-04-08 20:56:24,482 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 8fd130d9f8ee4bac98c987b38e0b4577, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1680987383903 2023-04-08 20:56:24,495 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): ebdeb8625c2b4ac57e00f30df2ca3028#B#compaction#242 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:56:24,503 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): ebdeb8625c2b4ac57e00f30df2ca3028#A#compaction#243 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:56:24,538 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:24,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:24,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:24,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:24,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:24,557 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:24,557 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:24,557 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:24,557 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:24,558 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,558 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:24,558 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:24,558 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:24,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:24,559 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,559 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:24,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:24,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:24,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:24,560 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:24,562 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:24,562 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:24,562 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:24,562 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:24,562 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:24,562 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:24,562 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:24,562 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:24,562 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:24,563 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ebdeb8625c2b4ac57e00f30df2ca3028 3/3 column families, dataSize=126.28 KB heapSize=362.32 KB 2023-04-08 20:56:24,563 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=A 2023-04-08 20:56:24,563 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:56:24,563 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=B 2023-04-08 20:56:24,563 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:56:24,563 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=C 2023-04-08 20:56:24,563 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:56:24,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on ebdeb8625c2b4ac57e00f30df2ca3028 2023-04-08 20:56:24,575 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:24,623 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:24,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987444618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,650 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=204 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/1be3c4f2b9de4e638a806ba98c4af784 2023-04-08 20:56:24,721 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=204 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/45b7a8ff355440cd9ad18b9a41445ced 2023-04-08 20:56:24,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:24,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987444724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,780 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=204 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/e6ed48e71d4c459a820bdc67a1fe2b8e 2023-04-08 20:56:24,784 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/1be3c4f2b9de4e638a806ba98c4af784 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/1be3c4f2b9de4e638a806ba98c4af784 2023-04-08 20:56:24,790 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/1be3c4f2b9de4e638a806ba98c4af784, entries=150, sequenceid=204, filesize=11.8 K 2023-04-08 20:56:24,792 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/45b7a8ff355440cd9ad18b9a41445ced as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/45b7a8ff355440cd9ad18b9a41445ced 2023-04-08 20:56:24,797 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/45b7a8ff355440cd9ad18b9a41445ced, entries=150, sequenceid=204, filesize=11.8 K 2023-04-08 20:56:24,799 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/e6ed48e71d4c459a820bdc67a1fe2b8e as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/e6ed48e71d4c459a820bdc67a1fe2b8e 2023-04-08 20:56:24,803 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/e6ed48e71d4c459a820bdc67a1fe2b8e, entries=150, sequenceid=204, filesize=11.8 K 2023-04-08 20:56:24,804 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~126.28 KB/129315, heapSize ~362.27 KB/370968, currentSize=53.67 KB/54960 for ebdeb8625c2b4ac57e00f30df2ca3028 in 242ms, sequenceid=204, compaction requested=true 2023-04-08 20:56:24,804 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:24,804 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:24,804 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:24,804 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:24,804 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,806 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:24,806 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:24,807 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:24,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:24,807 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:24,807 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,808 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:24,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:24,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:24,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:24,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:24,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5643af5c[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:24,809 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:24,809 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,810 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,810 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:24,810 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:24,810 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:24,811 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:24,811 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:24,811 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:24,815 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,815 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:24,815 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:24,815 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:24,815 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:24,815 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:24,815 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:24,815 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:24,822 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,822 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:24,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:24,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,822 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:24,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:24,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:24,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:24,823 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:24,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:24,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:24,825 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:24,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,830 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,830 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:24,830 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:24,830 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,830 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,830 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:24,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:24,830 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:24,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:24,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:24,830 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,830 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:24,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:24,830 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:24,831 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:24,831 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:24,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:24,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:24,931 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:24,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on ebdeb8625c2b4ac57e00f30df2ca3028 2023-04-08 20:56:24,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:24,933 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/dbc957444c8945cd8fe4635a56267437 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/dbc957444c8945cd8fe4635a56267437 2023-04-08 20:56:24,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:24,935 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ebdeb8625c2b4ac57e00f30df2ca3028 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-08 20:56:24,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:24,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:24,936 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:24,936 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:24,936 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:24,936 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:24,936 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=A 2023-04-08 20:56:24,936 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:24,936 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=B 2023-04-08 20:56:24,936 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:24,936 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=C 2023-04-08 20:56:24,936 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:24,937 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,937 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:24,938 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:24,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:24,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:24,938 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:24,939 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,939 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:24,939 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ebdeb8625c2b4ac57e00f30df2ca3028/A of ebdeb8625c2b4ac57e00f30df2ca3028 into dbc957444c8945cd8fe4635a56267437(size=12.1 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:56:24,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,939 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:24,939 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028., storeName=ebdeb8625c2b4ac57e00f30df2ca3028/A, priority=13, startTime=1680987384478; duration=0sec 2023-04-08 20:56:24,939 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:56:24,940 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-08 20:56:24,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:24,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:24,940 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:24,941 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60052 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-08 20:56:24,941 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): ebdeb8625c2b4ac57e00f30df2ca3028/C is initiating minor compaction (all files) 2023-04-08 20:56:24,941 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ebdeb8625c2b4ac57e00f30df2ca3028/C in TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:24,942 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/667d8062e47d4252b66948dfbd2fc0da, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/314678b7962340c3ba02e10fd0f7ed5b, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/f32dfaac5bfe4f0a8e2d669c264e6f84, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/7fa87866a6654accb0cf57a730b16e41, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/e6ed48e71d4c459a820bdc67a1fe2b8e] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp, totalSize=58.6 K 2023-04-08 20:56:24,942 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 667d8062e47d4252b66948dfbd2fc0da, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1680987380122 2023-04-08 20:56:24,942 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 314678b7962340c3ba02e10fd0f7ed5b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1680987381600 2023-04-08 20:56:24,943 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting f32dfaac5bfe4f0a8e2d669c264e6f84, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1680987381716 2023-04-08 20:56:24,946 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:24,946 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:24,946 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:24,946 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:24,947 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:24,947 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 7fa87866a6654accb0cf57a730b16e41, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1680987383903 2023-04-08 20:56:24,948 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting e6ed48e71d4c459a820bdc67a1fe2b8e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1680987384244 2023-04-08 20:56:24,951 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:24,951 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:24,951 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:24,951 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:24,951 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/ee3167aca4e549c6902d3e39c7458d40 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/ee3167aca4e549c6902d3e39c7458d40 2023-04-08 20:56:24,951 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:24,951 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:24,951 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:24,952 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:24,952 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,952 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:24,952 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:24,955 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,955 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:24,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:24,955 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,955 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:24,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:24,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:24,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:24,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:24,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:24,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75700cac[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:24,957 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:24,957 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,958 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,958 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,958 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:24,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,958 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:24,958 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:24,958 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:24,959 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:24,959 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:24,959 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,959 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:24,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:24,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:24,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:24,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:24,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:24,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:24,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:24,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:24,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:24,961 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:24,961 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:24,961 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:24,961 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:24,961 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:24,962 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,962 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:24,962 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:24,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:24,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:24,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:24,963 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:24,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:24,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:24,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:24,965 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,966 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): ebdeb8625c2b4ac57e00f30df2ca3028#C#compaction#248 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:56:24,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:24,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:24,967 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,967 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:24,967 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:24,967 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:24,968 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:24,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:24,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:24,968 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:24,968 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,968 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:24,968 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:24,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:24,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:24,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:24,969 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ebdeb8625c2b4ac57e00f30df2ca3028/B of ebdeb8625c2b4ac57e00f30df2ca3028 into ee3167aca4e549c6902d3e39c7458d40(size=12.1 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:56:24,969 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:24,969 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028., storeName=ebdeb8625c2b4ac57e00f30df2ca3028/B, priority=13, startTime=1680987384478; duration=0sec 2023-04-08 20:56:24,969 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:56:24,971 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:24,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:24,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:24,971 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:24,971 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:24,990 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/2839ef00ae7e4be4a1dfb6483b9a9fa9 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/2839ef00ae7e4be4a1dfb6483b9a9fa9 2023-04-08 20:56:24,994 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ebdeb8625c2b4ac57e00f30df2ca3028/C of ebdeb8625c2b4ac57e00f30df2ca3028 into 2839ef00ae7e4be4a1dfb6483b9a9fa9(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:56:24,994 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:24,994 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028., storeName=ebdeb8625c2b4ac57e00f30df2ca3028/C, priority=11, startTime=1680987384479; duration=0sec 2023-04-08 20:56:24,995 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:56:25,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:25,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987445035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,071 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:25,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:25,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:25,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:25,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:25,074 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:25,074 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:25,074 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:25,075 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:25,075 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,075 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:25,076 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:25,076 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:25,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:25,076 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,076 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:25,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:25,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:25,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:25,077 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:25,094 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:25,094 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:25,094 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:25,094 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:25,094 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:25,098 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:25,098 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:25,098 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:25,098 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:25,098 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:25,098 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:25,098 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:25,098 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:25,098 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,098 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:25,098 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:25,099 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:25,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:25,100 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,100 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,100 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:25,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:25,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:25,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:25,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:25,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:25,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c0e2282[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:25,101 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:25,101 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,102 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,102 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:25,102 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:25,102 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:25,102 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,102 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:25,103 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:25,103 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:25,104 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:25,104 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:25,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:25,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:25,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:25,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:25,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:25,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:25,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:25,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:25,108 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:25,108 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:25,108 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:25,108 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:25,108 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:25,113 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,113 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:25,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,113 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:25,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:25,114 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:25,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:25,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:25,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:25,117 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:25,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,126 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:25,126 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:25,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:25,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:25,127 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:25,127 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:25,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:25,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:25,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:25,127 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:25,127 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:25,127 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:25,127 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:25,127 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,127 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,127 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,127 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,140 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:25,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987445139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,227 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:25,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:25,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:25,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:25,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:25,242 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:25,242 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:25,242 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:25,242 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:25,243 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,243 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:25,244 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,244 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:25,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:25,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:25,244 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:25,244 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:25,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:25,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:25,245 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:25,250 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:25,250 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:25,250 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:25,250 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:25,250 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:25,258 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:25,258 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:25,258 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:25,258 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:25,258 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:25,258 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:25,258 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:25,258 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:25,258 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,258 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:25,258 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:25,259 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:25,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:25,259 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:25,260 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,260 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:25,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:25,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:25,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:25,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:25,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b2f971c[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:25,261 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:25,261 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,262 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,262 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:25,262 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:25,262 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:25,262 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,262 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:25,263 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,263 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:25,263 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:25,263 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:25,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:25,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:25,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:25,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:25,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:25,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:25,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:25,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:25,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:25,265 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:25,265 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:25,265 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:25,265 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:25,265 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:25,267 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,267 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,267 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:25,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:25,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:25,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:25,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:25,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:25,268 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:25,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:25,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:25,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,270 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:25,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:25,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,271 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,271 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:25,271 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:25,271 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:25,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:25,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:25,271 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:25,271 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,272 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,272 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,272 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,271 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:25,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:25,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:25,272 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:25,272 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:25,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:25,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:25,345 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:25,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987445343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,366 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/81e85d1b5e214c4595ad37575b5e145d 2023-04-08 20:56:25,372 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:25,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:25,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:25,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:25,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:25,398 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:25,398 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:25,398 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:25,398 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:25,399 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,399 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:25,399 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:25,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:25,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:25,399 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:25,400 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,400 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:25,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:25,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:25,400 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:25,417 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:25,417 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:25,417 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:25,417 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:25,417 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:25,420 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:25,420 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:25,420 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:25,420 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:25,420 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:25,420 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:25,420 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:25,420 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:25,420 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,420 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:25,420 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:25,421 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,421 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:25,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:25,422 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:25,422 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:25,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:25,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:25,425 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/d735dee4216842d9aacf3a512ca201fc 2023-04-08 20:56:25,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:25,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:25,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@505df26a[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:25,429 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:25,429 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,436 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,437 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:25,437 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:25,437 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:25,438 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,438 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:25,438 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:25,438 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:25,438 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:25,439 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:25,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:25,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:25,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:25,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:25,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:25,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:25,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:25,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:25,444 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:25,444 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:25,444 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:25,444 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:25,444 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:25,451 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,451 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,451 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:25,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:25,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:25,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:25,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:25,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,455 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:25,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:25,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:25,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:25,459 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:25,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:25,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,464 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:25,464 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:25,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:25,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:25,465 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,465 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:25,465 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:25,465 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:25,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:25,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:25,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:25,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:25,465 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:25,465 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,465 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,465 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,465 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:25,465 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:25,566 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:25,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:25,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:25,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:25,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:25,570 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:25,570 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:25,570 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:25,570 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:25,570 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,570 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:25,571 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:25,571 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:25,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:25,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:25,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,572 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,572 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:25,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:25,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:25,572 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:25,578 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:25,578 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:25,578 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:25,578 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:25,578 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:25,582 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:25,582 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:25,582 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:25,582 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:25,582 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:25,582 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:25,582 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:25,582 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:25,582 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,582 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:25,582 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:25,588 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,588 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:25,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:25,588 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,588 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:25,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:25,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:25,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:25,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:25,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:25,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@253d91a1[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:25,590 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:25,590 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,593 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,593 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:25,593 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:25,593 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:25,593 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,593 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:25,594 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:25,594 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:25,594 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,594 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:25,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:25,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:25,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:25,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:25,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:25,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:25,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:25,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:25,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:25,596 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:25,596 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:25,596 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:25,596 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:25,596 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:25,597 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,597 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:25,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:25,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:25,597 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:25,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:25,597 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:25,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:25,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:25,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:25,598 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:25,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:25,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,599 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,599 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:25,599 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:25,599 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:25,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:25,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:25,599 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:25,599 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:25,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:25,600 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:25,600 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:25,600 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,600 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,600 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:25,600 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:25,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:25,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:25,650 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:25,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987445648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,700 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:25,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:25,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:25,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:25,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:25,707 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:25,707 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:25,707 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:25,707 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:25,707 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,707 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:25,708 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,708 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:25,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:25,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:25,708 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,708 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:25,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:25,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:25,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:25,709 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:25,709 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:25,709 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:25,709 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:25,709 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:25,709 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:25,709 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:25,709 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:25,709 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:25,709 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:25,709 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:25,709 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:25,709 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:25,709 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:25,709 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,709 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:25,709 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:25,710 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,710 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:25,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:25,710 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,710 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:25,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:25,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:25,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:25,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:25,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:25,712 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:25,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@686aadb6[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:25,712 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,712 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,712 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:25,712 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:25,712 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,712 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:25,712 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:25,713 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:25,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:25,713 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:25,713 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:25,713 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:25,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:25,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:25,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:25,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:25,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:25,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:25,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:25,715 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:25,715 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:25,715 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:25,715 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:25,715 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:25,716 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:25,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:25,716 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,716 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:25,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:25,716 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:25,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:25,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:25,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:25,717 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:25,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:25,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,718 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,719 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:25,719 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:25,719 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:25,719 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:25,719 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:25,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:25,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:25,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:25,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:25,719 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,719 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,719 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:25,719 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:25,719 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:25,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:25,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:25,719 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,819 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:25,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:25,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:25,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:25,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:25,827 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:25,827 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:25,827 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:25,827 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:25,828 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,828 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:25,829 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:25,829 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:25,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:25,830 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,830 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:25,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:25,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:25,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:25,832 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:25,832 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:25,832 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:25,832 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:25,832 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:25,832 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:25,833 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:25,833 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:25,833 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:25,833 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:25,833 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:25,833 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:25,833 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:25,833 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:25,833 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,833 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:25,833 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:25,838 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:25,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:25,838 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,839 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,839 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:25,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:25,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:25,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:25,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:25,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:25,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f1a50d0[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:25,841 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:25,841 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,843 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,843 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:25,843 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:25,843 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:25,844 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,844 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:25,844 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:25,844 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:25,844 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,844 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:25,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:25,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:25,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:25,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:25,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:25,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:25,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:25,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:25,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:25,848 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:25,848 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:25,848 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:25,848 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:25,848 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:25,849 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,849 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:25,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,849 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:25,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:25,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:25,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:25,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:25,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,851 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:25,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:25,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:25,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,852 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:25,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:25,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,854 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,854 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:25,854 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:25,854 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:25,854 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,854 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:25,854 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:25,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:25,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:25,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:25,854 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:25,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:25,854 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:25,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:25,855 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,855 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:25,855 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:25,910 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/e25c9557abab4a3cbffcc240a86bacc1 2023-04-08 20:56:25,914 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/81e85d1b5e214c4595ad37575b5e145d as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/81e85d1b5e214c4595ad37575b5e145d 2023-04-08 20:56:25,917 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/81e85d1b5e214c4595ad37575b5e145d, entries=200, sequenceid=217, filesize=14.1 K 2023-04-08 20:56:25,918 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/d735dee4216842d9aacf3a512ca201fc as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/d735dee4216842d9aacf3a512ca201fc 2023-04-08 20:56:25,920 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/d735dee4216842d9aacf3a512ca201fc, entries=150, sequenceid=217, filesize=11.8 K 2023-04-08 20:56:25,921 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/e25c9557abab4a3cbffcc240a86bacc1 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/e25c9557abab4a3cbffcc240a86bacc1 2023-04-08 20:56:25,924 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/e25c9557abab4a3cbffcc240a86bacc1, entries=150, sequenceid=217, filesize=11.8 K 2023-04-08 20:56:25,925 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for ebdeb8625c2b4ac57e00f30df2ca3028 in 990ms, sequenceid=217, compaction requested=true 2023-04-08 20:56:25,925 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:25,925 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:56:25,925 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:56:25,925 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:56:25,925 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-08 20:56:25,926 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:56:25,926 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38840 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:56:25,926 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): ebdeb8625c2b4ac57e00f30df2ca3028/A is initiating minor compaction (all files) 2023-04-08 20:56:25,926 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:56:25,926 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ebdeb8625c2b4ac57e00f30df2ca3028/A in TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:25,926 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): ebdeb8625c2b4ac57e00f30df2ca3028/B is initiating minor compaction (all files) 2023-04-08 20:56:25,927 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/dbc957444c8945cd8fe4635a56267437, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/1be3c4f2b9de4e638a806ba98c4af784, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/81e85d1b5e214c4595ad37575b5e145d] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp, totalSize=37.9 K 2023-04-08 20:56:25,927 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ebdeb8625c2b4ac57e00f30df2ca3028/B in TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:25,927 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/ee3167aca4e549c6902d3e39c7458d40, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/45b7a8ff355440cd9ad18b9a41445ced, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/d735dee4216842d9aacf3a512ca201fc] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp, totalSize=35.6 K 2023-04-08 20:56:25,927 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting dbc957444c8945cd8fe4635a56267437, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1680987383903 2023-04-08 20:56:25,927 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting ee3167aca4e549c6902d3e39c7458d40, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1680987383903 2023-04-08 20:56:25,927 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 1be3c4f2b9de4e638a806ba98c4af784, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1680987384244 2023-04-08 20:56:25,927 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 45b7a8ff355440cd9ad18b9a41445ced, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1680987384244 2023-04-08 20:56:25,928 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 81e85d1b5e214c4595ad37575b5e145d, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1680987384598 2023-04-08 20:56:25,928 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting d735dee4216842d9aacf3a512ca201fc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1680987384617 2023-04-08 20:56:25,933 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): ebdeb8625c2b4ac57e00f30df2ca3028#A#compaction#251 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:56:25,937 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): ebdeb8625c2b4ac57e00f30df2ca3028#B#compaction#252 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:56:25,955 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:25,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:25,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:25,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:25,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:25,966 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:25,966 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:25,966 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:25,966 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:25,966 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,967 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:25,967 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:25,967 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:25,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:25,968 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:25,968 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:25,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:25,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:25,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:25,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:25,969 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:25,969 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:25,969 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:25,969 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:25,969 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:25,969 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:25,970 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:25,970 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:25,970 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:25,970 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:25,970 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ebdeb8625c2b4ac57e00f30df2ca3028 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-08 20:56:25,971 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=A 2023-04-08 20:56:25,971 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/015077187ac54cf48bab725a8e89b705 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/015077187ac54cf48bab725a8e89b705 2023-04-08 20:56:25,971 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:25,971 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=B 2023-04-08 20:56:25,971 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:25,971 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=C 2023-04-08 20:56:25,971 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:25,975 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ebdeb8625c2b4ac57e00f30df2ca3028/A of ebdeb8625c2b4ac57e00f30df2ca3028 into 015077187ac54cf48bab725a8e89b705(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:56:25,975 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:25,975 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028., storeName=ebdeb8625c2b4ac57e00f30df2ca3028/A, priority=13, startTime=1680987385925; duration=0sec 2023-04-08 20:56:25,975 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:56:25,976 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-08 20:56:25,976 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-08 20:56:25,976 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-08 20:56:25,976 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. because compaction request was cancelled 2023-04-08 20:56:25,986 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/ea88b23bcb0f401e9e5461d707f55f05 2023-04-08 20:56:26,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on ebdeb8625c2b4ac57e00f30df2ca3028 2023-04-08 20:56:26,157 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:26,208 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:26,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987446205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:26,310 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:26,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987446309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:26,365 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/99fad3c46b484d2b88474a22f8b40109 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/99fad3c46b484d2b88474a22f8b40109 2023-04-08 20:56:26,371 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ebdeb8625c2b4ac57e00f30df2ca3028/B of ebdeb8625c2b4ac57e00f30df2ca3028 into 99fad3c46b484d2b88474a22f8b40109(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:56:26,371 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:26,371 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028., storeName=ebdeb8625c2b4ac57e00f30df2ca3028/B, priority=13, startTime=1680987385925; duration=0sec 2023-04-08 20:56:26,371 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:56:26,415 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/90f0baed13a341248a7a7db09aa53731 2023-04-08 20:56:26,511 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:26,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987446511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:26,814 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:26,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987446813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:26,840 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/db04c203fb8545d3a10313f0972a119d 2023-04-08 20:56:26,844 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/ea88b23bcb0f401e9e5461d707f55f05 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/ea88b23bcb0f401e9e5461d707f55f05 2023-04-08 20:56:26,847 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/ea88b23bcb0f401e9e5461d707f55f05, entries=150, sequenceid=239, filesize=11.8 K 2023-04-08 20:56:26,848 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/90f0baed13a341248a7a7db09aa53731 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/90f0baed13a341248a7a7db09aa53731 2023-04-08 20:56:26,852 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/90f0baed13a341248a7a7db09aa53731, entries=150, sequenceid=239, filesize=11.8 K 2023-04-08 20:56:26,856 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/db04c203fb8545d3a10313f0972a119d as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/db04c203fb8545d3a10313f0972a119d 2023-04-08 20:56:26,860 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/db04c203fb8545d3a10313f0972a119d, entries=150, sequenceid=239, filesize=11.8 K 2023-04-08 20:56:26,861 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for ebdeb8625c2b4ac57e00f30df2ca3028 in 891ms, sequenceid=239, compaction requested=true 2023-04-08 20:56:26,861 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:26,861 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:26,861 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:26,861 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:26,861 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:26,861 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:26,861 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:26,863 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:26,863 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:26,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:26,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:26,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:26,863 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:26,863 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:26,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:26,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:26,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:26,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:26,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:26,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:26,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30eb3dfb[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:26,865 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:26,865 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:26,866 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:26,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:26,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:26,866 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:26,866 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:26,866 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:26,866 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:26,866 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:26,867 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:26,867 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:26,867 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:26,867 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:26,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:26,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:26,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:26,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:26,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:26,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:26,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:26,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:26,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:26,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:26,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:26,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:26,869 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:26,869 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:26,869 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:26,869 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:26,870 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:26,871 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:26,871 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:26,871 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:26,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:26,871 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:26,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:26,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:26,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:26,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:26,872 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:26,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:26,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:26,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:26,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:26,874 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:26,874 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:26,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:26,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:26,875 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:26,875 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:26,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:26,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:26,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:26,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:26,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:26,875 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:26,875 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:26,875 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:26,875 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:26,875 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:26,875 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:26,875 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:26,875 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:26,975 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:26,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:26,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:26,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:26,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:26,979 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:26,979 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:26,979 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:26,979 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:26,980 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:26,980 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:26,980 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:26,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:26,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:26,980 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:26,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:26,981 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:26,981 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:26,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:26,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:26,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:26,981 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:26,981 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:26,982 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:26,982 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:26,982 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:26,982 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:26,983 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:26,983 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:26,983 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:26,983 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:26,983 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ebdeb8625c2b4ac57e00f30df2ca3028 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-08 20:56:26,983 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=A 2023-04-08 20:56:26,983 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:26,983 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=B 2023-04-08 20:56:26,984 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:26,984 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=C 2023-04-08 20:56:26,984 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:27,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on ebdeb8625c2b4ac57e00f30df2ca3028 2023-04-08 20:56:27,318 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:27,364 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:27,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987447362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:27,400 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/e9858080f6e941c5b59f996e82d0c443 2023-04-08 20:56:27,466 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:27,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987447465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:27,668 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:27,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987447667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:27,809 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/ddf855bf797247f69ad8685dc61cefd6 2023-04-08 20:56:27,857 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/334261fcc3c54c62bc6de484c8b69b1d 2023-04-08 20:56:27,863 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/e9858080f6e941c5b59f996e82d0c443 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/e9858080f6e941c5b59f996e82d0c443 2023-04-08 20:56:27,867 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/e9858080f6e941c5b59f996e82d0c443, entries=150, sequenceid=254, filesize=11.8 K 2023-04-08 20:56:27,870 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/ddf855bf797247f69ad8685dc61cefd6 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/ddf855bf797247f69ad8685dc61cefd6 2023-04-08 20:56:27,875 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/ddf855bf797247f69ad8685dc61cefd6, entries=150, sequenceid=254, filesize=11.8 K 2023-04-08 20:56:27,876 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/334261fcc3c54c62bc6de484c8b69b1d as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/334261fcc3c54c62bc6de484c8b69b1d 2023-04-08 20:56:27,881 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/334261fcc3c54c62bc6de484c8b69b1d, entries=150, sequenceid=254, filesize=11.8 K 2023-04-08 20:56:27,882 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for ebdeb8625c2b4ac57e00f30df2ca3028 in 899ms, sequenceid=254, compaction requested=true 2023-04-08 20:56:27,882 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:27,882 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:27,882 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:27,883 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:27,883 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:27,883 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:27,883 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:27,884 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:27,884 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:27,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:27,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:27,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:27,884 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:27,884 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:27,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:27,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:27,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:27,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:27,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:27,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:27,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@791212de[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:27,886 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:27,886 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:27,887 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:27,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:27,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:27,887 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:27,887 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:27,887 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:27,887 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:27,887 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:27,888 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:27,888 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:27,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:27,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:27,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:27,888 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:27,888 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:27,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:27,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:27,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:27,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:27,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:27,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:27,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:27,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:27,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:27,890 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:27,890 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:27,890 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:27,890 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:27,890 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:27,891 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:27,891 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:27,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:27,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:27,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:27,891 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:27,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:27,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:27,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:27,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:27,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:27,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:27,892 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:27,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:27,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:27,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:27,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:27,893 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:27,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:27,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:27,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:27,895 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:27,895 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:27,895 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:27,895 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:27,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:27,895 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:27,895 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:27,895 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:27,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:27,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:27,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:27,895 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:27,895 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:27,896 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:27,895 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:27,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:27,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:27,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:27,895 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:27,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on ebdeb8625c2b4ac57e00f30df2ca3028 2023-04-08 20:56:27,972 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ebdeb8625c2b4ac57e00f30df2ca3028 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-08 20:56:27,972 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=A 2023-04-08 20:56:27,972 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:27,972 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=B 2023-04-08 20:56:27,972 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:27,972 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=C 2023-04-08 20:56:27,972 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:27,996 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:27,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:27,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:27,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:27,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:27,998 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:27,998 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:27,998 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:27,998 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:27,999 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:27,999 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:27,999 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:27,999 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:27,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:28,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:28,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:28,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:28,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:28,000 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:28,000 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:28,001 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:28,001 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:28,001 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:28,001 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:28,001 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:28,001 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:28,001 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:28,001 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:28,001 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:28,001 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:28,001 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:28,001 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:28,001 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,001 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,001 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:28,001 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:28,001 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:28,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:28,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987447999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,002 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,002 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:28,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:28,002 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,002 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:28,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:28,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:28,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:28,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:28,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44a80757[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:28,003 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:28,003 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,004 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,004 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:28,004 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:28,004 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:28,004 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,004 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:28,005 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,005 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:28,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:28,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:28,005 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:28,005 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:28,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:28,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:28,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:28,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:28,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:28,006 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:28,007 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:28,007 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:28,007 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:28,007 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:28,008 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,008 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:28,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,008 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:28,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:28,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:28,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:28,009 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:28,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,010 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:28,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:28,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,011 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:28,011 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:28,011 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:28,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:28,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:28,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:28,012 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:28,011 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:28,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:28,012 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:28,012 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,012 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:28,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:28,012 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:28,012 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:28,012 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,012 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:28,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987448102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,112 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:28,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:28,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:28,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:28,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:28,115 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:28,115 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:28,115 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:28,115 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:28,115 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,115 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:28,116 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:28,116 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:28,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:28,116 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,116 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:28,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:28,116 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:28,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:28,117 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:28,117 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:28,117 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:28,117 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:28,117 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:28,117 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:28,117 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:28,117 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:28,117 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:28,117 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:28,117 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:28,117 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:28,117 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:28,117 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:28,117 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,117 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:28,117 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:28,118 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:28,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:28,118 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:28,119 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,119 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:28,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:28,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:28,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:28,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19e953c2[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:28,120 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:28,120 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,121 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,121 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:28,121 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,121 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:28,121 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:28,121 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:28,122 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,122 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:28,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:28,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:28,122 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:28,122 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:28,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:28,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:28,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:28,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:28,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:28,124 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:28,124 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:28,124 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:28,124 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:28,124 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:28,125 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,125 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:28,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:28,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,125 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:28,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:28,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:28,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:28,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,126 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:28,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,127 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:28,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,128 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,128 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:28,128 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:28,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:28,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:28,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:28,129 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:28,129 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:28,129 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,129 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,129 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,129 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:28,129 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:28,129 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:28,129 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:28,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:28,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:28,229 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:28,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:28,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:28,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:28,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:28,232 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:28,232 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:28,232 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:28,232 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:28,232 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,232 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:28,233 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,233 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:28,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:28,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:28,233 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,233 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:28,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:28,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:28,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:28,235 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:28,235 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:28,235 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:28,235 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:28,235 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:28,235 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:28,235 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:28,235 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:28,235 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:28,236 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:28,236 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:28,236 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:28,236 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:28,236 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:28,236 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,236 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:28,236 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:28,237 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,237 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:28,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:28,237 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,237 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:28,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:28,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:28,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:28,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:28,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7084b1e1[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:28,238 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:28,238 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,238 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,238 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:28,239 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:28,239 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:28,239 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,239 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:28,239 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,239 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:28,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:28,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:28,239 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:28,239 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:28,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:28,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:28,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:28,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:28,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:28,241 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:28,241 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:28,241 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:28,241 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:28,241 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:28,242 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,242 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,242 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:28,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:28,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:28,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:28,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:28,243 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:28,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:28,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,243 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:28,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,245 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:28,245 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:28,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:28,245 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:28,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:28,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:28,245 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:28,245 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:28,245 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:28,245 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:28,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:28,245 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,245 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,245 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,245 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:28,245 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:28,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:28,280 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:28,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44412 deadline: 1680987448277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,281 DEBUG [Thread-1669] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8157 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028., hostname=jenkins-hbase20.apache.org,41091,1680987313601, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 20:56:28,307 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:28,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44428 deadline: 1680987448304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,307 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:28,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987448305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,308 DEBUG [Thread-1665] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8182 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028., hostname=jenkins-hbase20.apache.org,41091,1680987313601, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 20:56:28,312 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:28,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44448 deadline: 1680987448309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,314 DEBUG [Thread-1661] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8189 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028., hostname=jenkins-hbase20.apache.org,41091,1680987313601, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 20:56:28,317 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:28,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44430 deadline: 1680987448316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,318 DEBUG [Thread-1663] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8191 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028., hostname=jenkins-hbase20.apache.org,41091,1680987313601, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 20:56:28,346 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:28,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:28,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:28,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:28,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:28,348 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:28,348 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:28,348 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:28,348 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:28,349 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,349 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:28,349 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:28,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:28,349 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:28,350 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,350 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:28,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:28,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:28,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:28,350 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:28,351 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:28,351 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:28,351 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:28,351 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:28,351 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:28,351 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:28,351 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:28,351 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:28,351 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:28,351 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:28,351 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:28,351 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:28,351 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:28,351 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,351 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:28,351 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:28,352 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,352 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:28,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:28,352 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,352 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:28,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:28,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:28,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:28,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:28,353 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:28,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19e9d9cd[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:28,353 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,354 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,354 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:28,354 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:28,354 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:28,354 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,354 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:28,355 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:28,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:28,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:28,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:28,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,356 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:28,356 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:28,356 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:28,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:28,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:28,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:28,357 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:28,357 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:28,357 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:28,357 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:28,357 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:28,358 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,358 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:28,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:28,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,358 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:28,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:28,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:28,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:28,359 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:28,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,360 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:28,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,361 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,361 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:28,361 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:28,361 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:28,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:28,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:28,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:28,361 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:28,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:28,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:28,361 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:28,361 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,361 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,361 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:28,361 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:28,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:28,361 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:28,361 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,379 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=274 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/42569f0ad01244b78e57474fbd8e7196 2023-04-08 20:56:28,391 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=274 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/2a1c135d15024dd2bb39145279b21470 2023-04-08 20:56:28,461 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:28,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:28,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:28,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:28,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:28,464 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:28,464 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:28,464 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:28,464 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:28,464 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,464 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:28,465 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,465 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:28,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:28,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:28,465 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,465 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:28,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:28,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:28,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:28,466 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:28,466 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:28,466 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:28,466 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:28,466 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:28,466 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:28,466 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:28,466 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:28,466 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:28,467 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:28,467 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:28,467 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:28,467 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:28,467 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:28,467 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,467 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:28,467 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:28,468 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,468 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:28,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:28,468 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,468 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:28,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:28,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:28,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:28,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:28,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dffe718[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:28,469 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:28,469 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,470 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,470 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:28,471 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,471 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:28,471 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:28,471 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:28,473 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,473 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:28,473 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:28,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:28,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:28,473 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:28,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:28,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:28,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:28,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:28,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:28,475 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:28,475 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:28,475 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:28,475 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:28,475 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:28,476 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,476 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:28,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:28,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,476 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:28,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:28,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:28,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:28,477 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:28,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,479 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:28,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,480 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,480 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:28,480 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:28,480 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,480 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,480 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:28,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:28,480 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:28,480 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:28,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:28,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:28,480 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:28,480 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:28,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:28,480 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,480 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:28,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:28,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:28,581 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:28,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:28,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:28,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:28,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:28,583 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:28,583 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:28,583 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:28,583 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:28,584 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,584 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:28,584 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,584 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:28,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:28,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:28,585 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,585 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:28,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:28,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:28,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:28,585 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:28,586 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:28,586 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:28,586 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:28,586 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:28,586 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:28,586 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:28,586 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:28,586 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:28,586 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:28,586 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:28,587 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:28,587 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:28,587 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:28,587 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,587 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:28,587 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:28,587 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,587 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:28,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:28,588 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,588 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:28,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:28,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:28,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:28,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:28,589 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:28,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4edbd026[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:28,589 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,589 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,589 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:28,590 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:28,590 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,590 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:28,590 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:28,590 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:28,590 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,590 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:28,590 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:28,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:28,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:28,592 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:28,592 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:28,592 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:28,592 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:28,592 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:28,593 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,593 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:28,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,593 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:28,594 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:28,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,595 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,596 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,596 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:28,596 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:28,597 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:28,597 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,597 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,597 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:28,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:28,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:28,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:28,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:28,597 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,597 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:28,597 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:28,597 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:28,597 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:28,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:28,611 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:28,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987448610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,697 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:28,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:28,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:28,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:28,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:28,700 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:28,700 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:28,700 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:28,700 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:28,700 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,701 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:28,701 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,701 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:28,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:28,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:28,701 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,702 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:28,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:28,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:28,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:28,702 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:28,702 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:28,702 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:28,702 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:28,702 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:28,702 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:28,702 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:28,702 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:28,703 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:28,703 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:28,703 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:28,703 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:28,703 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:28,703 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:28,703 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,703 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:28,703 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:28,703 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,703 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:28,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:28,704 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,704 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:28,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:28,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:28,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:28,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:28,705 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:28,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b856ba5[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:28,705 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,705 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,705 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:28,705 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:28,705 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,706 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:28,705 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:28,706 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:28,706 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:28,706 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,706 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:28,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:28,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:28,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:28,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:28,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:28,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:28,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:28,708 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:28,708 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:28,708 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:28,708 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:28,708 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:28,709 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,709 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:28,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,709 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:28,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:28,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:28,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:28,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:28,710 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:28,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,710 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:28,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:28,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,712 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,712 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:28,712 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:28,712 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:28,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:28,712 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:28,712 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:28,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:28,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:28,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:28,712 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,712 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,712 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:28,712 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:28,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:28,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:28,712 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:28,712 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:28,803 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=274 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/cdc4f6ac56fb4349a011c0d431fd5368 2023-04-08 20:56:28,807 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/42569f0ad01244b78e57474fbd8e7196 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/42569f0ad01244b78e57474fbd8e7196 2023-04-08 20:56:28,810 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/42569f0ad01244b78e57474fbd8e7196, entries=150, sequenceid=274, filesize=11.9 K 2023-04-08 20:56:28,811 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/2a1c135d15024dd2bb39145279b21470 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/2a1c135d15024dd2bb39145279b21470 2023-04-08 20:56:28,812 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:28,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:28,814 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/2a1c135d15024dd2bb39145279b21470, entries=150, sequenceid=274, filesize=11.9 K 2023-04-08 20:56:28,814 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/cdc4f6ac56fb4349a011c0d431fd5368 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/cdc4f6ac56fb4349a011c0d431fd5368 2023-04-08 20:56:28,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:28,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:28,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:28,816 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:28,817 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:28,817 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:28,817 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:28,817 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,817 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:28,818 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,818 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:28,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:28,818 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/cdc4f6ac56fb4349a011c0d431fd5368, entries=150, sequenceid=274, filesize=11.9 K 2023-04-08 20:56:28,818 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:28,818 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:28,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:28,819 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for ebdeb8625c2b4ac57e00f30df2ca3028 in 846ms, sequenceid=274, compaction requested=true 2023-04-08 20:56:28,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:28,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:28,819 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:28,819 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:28,819 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:56:28,819 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 20:56:28,819 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 20:56:28,819 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:28,819 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:28,819 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:56:28,819 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:28,819 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:56:28,819 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:28,819 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:28,819 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:28,819 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:28,819 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:28,819 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:28,820 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ebdeb8625c2b4ac57e00f30df2ca3028 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-08 20:56:28,820 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=A 2023-04-08 20:56:28,820 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:28,820 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=B 2023-04-08 20:56:28,820 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:28,820 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=C 2023-04-08 20:56:28,820 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48738 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 20:56:28,820 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:28,820 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): ebdeb8625c2b4ac57e00f30df2ca3028/A is initiating minor compaction (all files) 2023-04-08 20:56:28,820 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ebdeb8625c2b4ac57e00f30df2ca3028/A in TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:28,820 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/015077187ac54cf48bab725a8e89b705, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/ea88b23bcb0f401e9e5461d707f55f05, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/e9858080f6e941c5b59f996e82d0c443, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/42569f0ad01244b78e57474fbd8e7196] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp, totalSize=47.6 K 2023-04-08 20:56:28,821 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48738 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 20:56:28,821 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): ebdeb8625c2b4ac57e00f30df2ca3028/B is initiating minor compaction (all files) 2023-04-08 20:56:28,821 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ebdeb8625c2b4ac57e00f30df2ca3028/B in TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:28,821 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/99fad3c46b484d2b88474a22f8b40109, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/90f0baed13a341248a7a7db09aa53731, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/ddf855bf797247f69ad8685dc61cefd6, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/2a1c135d15024dd2bb39145279b21470] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp, totalSize=47.6 K 2023-04-08 20:56:28,821 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 015077187ac54cf48bab725a8e89b705, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1680987384617 2023-04-08 20:56:28,822 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting ea88b23bcb0f401e9e5461d707f55f05, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1680987385024 2023-04-08 20:56:28,822 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 99fad3c46b484d2b88474a22f8b40109, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1680987384617 2023-04-08 20:56:28,822 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting e9858080f6e941c5b59f996e82d0c443, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1680987386194 2023-04-08 20:56:28,822 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 90f0baed13a341248a7a7db09aa53731, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1680987385024 2023-04-08 20:56:28,822 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 42569f0ad01244b78e57474fbd8e7196, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=274, earliestPutTs=1680987387355 2023-04-08 20:56:28,823 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting ddf855bf797247f69ad8685dc61cefd6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1680987386194 2023-04-08 20:56:28,823 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 2a1c135d15024dd2bb39145279b21470, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=274, earliestPutTs=1680987387355 2023-04-08 20:56:28,831 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): ebdeb8625c2b4ac57e00f30df2ca3028#B#compaction#263 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:56:28,834 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): ebdeb8625c2b4ac57e00f30df2ca3028#A#compaction#264 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:56:28,845 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/83e2f2aa1ecd4324b9465a8a099b845a as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/83e2f2aa1ecd4324b9465a8a099b845a 2023-04-08 20:56:28,848 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ebdeb8625c2b4ac57e00f30df2ca3028/A of ebdeb8625c2b4ac57e00f30df2ca3028 into 83e2f2aa1ecd4324b9465a8a099b845a(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:56:28,848 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:28,848 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028., storeName=ebdeb8625c2b4ac57e00f30df2ca3028/A, priority=12, startTime=1680987388819; duration=0sec 2023-04-08 20:56:28,848 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:56:28,848 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-08 20:56:28,850 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60672 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-08 20:56:28,850 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): ebdeb8625c2b4ac57e00f30df2ca3028/C is initiating minor compaction (all files) 2023-04-08 20:56:28,850 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ebdeb8625c2b4ac57e00f30df2ca3028/C in TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:28,850 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/2839ef00ae7e4be4a1dfb6483b9a9fa9, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/e25c9557abab4a3cbffcc240a86bacc1, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/db04c203fb8545d3a10313f0972a119d, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/334261fcc3c54c62bc6de484c8b69b1d, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/cdc4f6ac56fb4349a011c0d431fd5368] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp, totalSize=59.3 K 2023-04-08 20:56:28,850 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 2839ef00ae7e4be4a1dfb6483b9a9fa9, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1680987384244 2023-04-08 20:56:28,850 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting e25c9557abab4a3cbffcc240a86bacc1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1680987384617 2023-04-08 20:56:28,851 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting db04c203fb8545d3a10313f0972a119d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1680987385024 2023-04-08 20:56:28,851 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 334261fcc3c54c62bc6de484c8b69b1d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1680987386194 2023-04-08 20:56:28,851 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting cdc4f6ac56fb4349a011c0d431fd5368, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=274, earliestPutTs=1680987387355 2023-04-08 20:56:28,858 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): ebdeb8625c2b4ac57e00f30df2ca3028#C#compaction#265 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:56:28,864 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/c6aadbd82cb54dc4b86ab68af81766e5 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/c6aadbd82cb54dc4b86ab68af81766e5 2023-04-08 20:56:28,868 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ebdeb8625c2b4ac57e00f30df2ca3028/C of ebdeb8625c2b4ac57e00f30df2ca3028 into c6aadbd82cb54dc4b86ab68af81766e5(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:56:28,868 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:28,868 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028., storeName=ebdeb8625c2b4ac57e00f30df2ca3028/C, priority=11, startTime=1680987388819; duration=0sec 2023-04-08 20:56:28,868 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:56:29,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on ebdeb8625c2b4ac57e00f30df2ca3028 2023-04-08 20:56:29,114 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:29,161 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:29,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987449160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:29,231 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/788218e7680b42c98dd2c52211606911 2023-04-08 20:56:29,243 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/00f2634921d8410fb9224df22bf2bfbb as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/00f2634921d8410fb9224df22bf2bfbb 2023-04-08 20:56:29,246 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ebdeb8625c2b4ac57e00f30df2ca3028/B of ebdeb8625c2b4ac57e00f30df2ca3028 into 00f2634921d8410fb9224df22bf2bfbb(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:56:29,246 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:29,246 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028., storeName=ebdeb8625c2b4ac57e00f30df2ca3028/B, priority=12, startTime=1680987388819; duration=0sec 2023-04-08 20:56:29,246 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:56:29,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:29,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987449262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:29,466 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:29,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987449465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:29,644 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/e3990c72148940b9aa72053582507e8b 2023-04-08 20:56:29,769 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:29,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987449768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:30,058 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/9327d46b760749cf82d7fed39793361b 2023-04-08 20:56:30,061 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/788218e7680b42c98dd2c52211606911 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/788218e7680b42c98dd2c52211606911 2023-04-08 20:56:30,064 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/788218e7680b42c98dd2c52211606911, entries=150, sequenceid=286, filesize=11.9 K 2023-04-08 20:56:30,065 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/e3990c72148940b9aa72053582507e8b as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/e3990c72148940b9aa72053582507e8b 2023-04-08 20:56:30,068 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/e3990c72148940b9aa72053582507e8b, entries=150, sequenceid=286, filesize=11.9 K 2023-04-08 20:56:30,069 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/9327d46b760749cf82d7fed39793361b as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/9327d46b760749cf82d7fed39793361b 2023-04-08 20:56:30,072 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/9327d46b760749cf82d7fed39793361b, entries=150, sequenceid=286, filesize=11.9 K 2023-04-08 20:56:30,072 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for ebdeb8625c2b4ac57e00f30df2ca3028 in 1252ms, sequenceid=286, compaction requested=false 2023-04-08 20:56:30,072 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:30,072 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:30,073 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:30,073 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:30,073 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:30,073 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:30,073 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:30,073 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:30,073 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:30,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:30,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:30,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:30,074 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:30,074 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:30,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:30,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:30,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:30,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:30,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:30,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:30,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c75fea9[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:30,075 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:30,075 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:30,075 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:30,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:30,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:30,076 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:30,076 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:30,076 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:30,076 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:30,076 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:30,076 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:30,076 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:30,076 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:30,076 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:30,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:30,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:30,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:30,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:30,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:30,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:30,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:30,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:30,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:30,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:30,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:30,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:30,078 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:30,078 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:30,078 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:30,078 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:30,078 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:30,079 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:30,079 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:30,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:30,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:30,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:30,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:30,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:30,079 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:30,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:30,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:30,080 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:30,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:30,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:30,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:30,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:30,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:30,081 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:30,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:30,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:30,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:30,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:30,083 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:30,083 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:30,083 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:30,083 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:30,083 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:30,083 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:30,083 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:30,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:30,083 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:30,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:30,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:30,083 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:30,083 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:30,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:30,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:30,083 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:30,083 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:30,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:30,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:30,183 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:30,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:30,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:30,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:30,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:30,186 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:30,186 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:30,186 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:30,186 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:30,187 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:30,187 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:30,187 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:30,187 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:30,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:30,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:30,188 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:30,188 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:30,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:30,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:30,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:30,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:30,188 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:30,188 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:30,188 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:30,188 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:30,188 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:30,188 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:30,189 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:30,189 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:30,189 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:30,189 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:30,189 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ebdeb8625c2b4ac57e00f30df2ca3028 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-08 20:56:30,189 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=A 2023-04-08 20:56:30,189 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:30,189 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=B 2023-04-08 20:56:30,189 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:30,189 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=C 2023-04-08 20:56:30,189 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:30,194 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=309 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/cda2ad3a1fb74bb7b9577e75585ddcb9 2023-04-08 20:56:30,206 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=309 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/1e88adf60591498697b0a402312bec89 2023-04-08 20:56:30,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on ebdeb8625c2b4ac57e00f30df2ca3028 2023-04-08 20:56:30,276 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:30,297 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:30,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987450296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:30,399 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:30,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987450398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:30,602 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:30,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987450601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:30,619 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=309 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/ecb18c00d31c4b1a839eaf791be7ebfe 2023-04-08 20:56:30,622 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/cda2ad3a1fb74bb7b9577e75585ddcb9 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/cda2ad3a1fb74bb7b9577e75585ddcb9 2023-04-08 20:56:30,625 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/cda2ad3a1fb74bb7b9577e75585ddcb9, entries=150, sequenceid=309, filesize=11.9 K 2023-04-08 20:56:30,626 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/1e88adf60591498697b0a402312bec89 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/1e88adf60591498697b0a402312bec89 2023-04-08 20:56:30,630 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/1e88adf60591498697b0a402312bec89, entries=150, sequenceid=309, filesize=11.9 K 2023-04-08 20:56:30,630 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/ecb18c00d31c4b1a839eaf791be7ebfe as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/ecb18c00d31c4b1a839eaf791be7ebfe 2023-04-08 20:56:30,635 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/ecb18c00d31c4b1a839eaf791be7ebfe, entries=150, sequenceid=309, filesize=11.9 K 2023-04-08 20:56:30,636 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for ebdeb8625c2b4ac57e00f30df2ca3028 in 447ms, sequenceid=309, compaction requested=true 2023-04-08 20:56:30,636 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:30,636 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:30,636 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:30,636 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:30,636 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:30,638 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:30,638 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:30,639 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:30,639 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:30,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:30,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:30,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:30,639 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:30,639 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:30,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:30,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:30,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:30,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:30,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:30,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:30,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e500718[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:30,641 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:30,641 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:30,642 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:30,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:30,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:30,642 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:30,642 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:30,642 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:30,642 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:30,642 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:30,643 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:30,643 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:30,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:30,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:30,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:30,643 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:30,643 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:30,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:30,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:30,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:30,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:30,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:30,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:30,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:30,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:30,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:30,646 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:30,646 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:30,646 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:30,646 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:30,646 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:30,647 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:30,647 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:30,647 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:30,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:30,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:30,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:30,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:30,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:30,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:30,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:30,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:30,647 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:30,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:30,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:30,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:30,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:30,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:30,648 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:30,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:30,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:30,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:30,650 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:30,650 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:30,650 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:30,650 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:30,650 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:30,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:30,650 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:30,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:30,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:30,650 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:30,650 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:30,650 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:30,650 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:30,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:30,650 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:30,650 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:30,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:30,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:30,751 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:30,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:30,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:30,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:30,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:30,753 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:30,753 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:30,753 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:30,753 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:30,754 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:30,754 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:30,754 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:30,754 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:30,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:30,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:30,755 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:30,755 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:30,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:30,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:30,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:30,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:30,755 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:30,755 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:30,755 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:30,755 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:30,755 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:30,756 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:30,756 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:30,756 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:30,756 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:30,756 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:30,756 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ebdeb8625c2b4ac57e00f30df2ca3028 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-08 20:56:30,756 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=A 2023-04-08 20:56:30,756 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:30,756 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=B 2023-04-08 20:56:30,756 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:30,756 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=C 2023-04-08 20:56:30,756 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:30,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on ebdeb8625c2b4ac57e00f30df2ca3028 2023-04-08 20:56:30,905 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:30,964 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:30,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987450962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:31,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:31,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987451065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:31,162 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/3cf5514e5c074f3ebbba51ddf7513a54 2023-04-08 20:56:31,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:31,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987451267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:31,571 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/b10b0b4ec47f4669ae501870e28dc848 2023-04-08 20:56:31,572 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:31,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987451571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:31,979 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/5a64000fc6874832ba507f46d4d2f715 2023-04-08 20:56:31,983 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/3cf5514e5c074f3ebbba51ddf7513a54 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/3cf5514e5c074f3ebbba51ddf7513a54 2023-04-08 20:56:31,986 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/3cf5514e5c074f3ebbba51ddf7513a54, entries=100, sequenceid=321, filesize=9.5 K 2023-04-08 20:56:31,987 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/b10b0b4ec47f4669ae501870e28dc848 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/b10b0b4ec47f4669ae501870e28dc848 2023-04-08 20:56:31,989 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/b10b0b4ec47f4669ae501870e28dc848, entries=100, sequenceid=321, filesize=9.5 K 2023-04-08 20:56:31,990 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/5a64000fc6874832ba507f46d4d2f715 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/5a64000fc6874832ba507f46d4d2f715 2023-04-08 20:56:31,993 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/5a64000fc6874832ba507f46d4d2f715, entries=100, sequenceid=321, filesize=9.5 K 2023-04-08 20:56:31,994 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for ebdeb8625c2b4ac57e00f30df2ca3028 in 1238ms, sequenceid=321, compaction requested=true 2023-04-08 20:56:31,994 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:31,994 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:31,994 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:31,994 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:31,994 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:31,994 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:31,994 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:31,995 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:31,995 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:31,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:31,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:31,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:31,995 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:31,995 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:31,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:31,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:31,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:31,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:31,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:31,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:31,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e9ab814[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:31,996 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:31,997 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:31,997 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:31,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:31,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:31,997 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:31,997 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:31,997 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:31,997 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:31,997 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:31,998 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:31,998 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:31,998 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:31,998 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:31,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:31,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:31,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:31,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:31,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:31,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:31,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:31,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:31,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:31,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:32,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:32,000 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:32,000 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:32,000 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:32,000 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:32,000 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:32,000 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,000 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,001 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:32,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:32,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:32,001 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:32,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:32,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:32,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:32,002 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:32,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,003 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,003 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,003 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,003 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,003 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,003 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,004 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:32,004 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:32,004 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:32,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:32,003 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:32,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:32,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:32,004 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:32,004 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:32,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:32,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:32,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:32,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on ebdeb8625c2b4ac57e00f30df2ca3028 2023-04-08 20:56:32,078 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ebdeb8625c2b4ac57e00f30df2ca3028 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-08 20:56:32,078 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=A 2023-04-08 20:56:32,078 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:56:32,079 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=B 2023-04-08 20:56:32,079 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:56:32,079 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=C 2023-04-08 20:56:32,079 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:56:32,104 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:32,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:32,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:32,107 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:32,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987452104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:32,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:32,107 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:32,107 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:32,107 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:32,107 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:32,107 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,107 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,108 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,108 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:32,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:32,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:32,108 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,108 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:32,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:32,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,109 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:32,109 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:32,109 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:32,109 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:32,109 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:32,109 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:32,109 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:32,109 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:32,109 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:32,109 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:32,109 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:32,110 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:32,110 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:32,110 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:32,110 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,110 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:32,110 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:32,111 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,111 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:32,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:32,111 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,111 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:32,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:32,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:32,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:32,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:32,112 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:32,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e723421[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:32,112 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,113 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,113 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:32,113 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:32,113 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:32,113 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,113 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:32,113 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:32,113 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:32,113 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,113 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:32,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:32,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:32,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:32,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:32,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:32,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:32,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:32,115 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:32,116 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:32,116 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:32,116 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:32,116 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:32,116 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:32,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:32,116 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,117 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:32,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:32,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:32,117 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:32,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:32,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,118 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:32,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,120 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,120 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,120 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:32,120 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:32,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:32,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:32,120 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:32,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:32,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:32,120 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:32,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:32,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:32,120 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,120 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,120 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:32,120 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,121 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:32,121 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:32,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987452207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,221 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:32,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:32,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:32,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:32,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:32,224 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:32,224 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:32,224 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:32,224 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:32,224 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,224 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,225 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:32,225 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:32,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:32,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,225 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,225 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:32,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:32,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,226 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:32,226 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:32,226 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:32,226 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:32,226 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:32,226 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:32,226 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:32,226 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:32,226 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:32,226 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:32,226 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:32,226 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:32,226 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:32,227 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:32,227 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,227 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:32,227 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:32,227 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,227 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:32,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:32,228 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,228 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:32,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:32,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:32,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:32,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:32,229 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:32,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d9113f0[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:32,229 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,246 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,247 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:32,247 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:32,247 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:32,247 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,247 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:32,251 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,251 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:32,251 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:32,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:32,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:32,251 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:32,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:32,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:32,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:32,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:32,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:32,254 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:32,254 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:32,254 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:32,254 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:32,254 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:32,255 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,255 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:32,255 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:32,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:32,256 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:32,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:32,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:32,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,257 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:32,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,259 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:32,259 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:32,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:32,259 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:32,259 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:32,259 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,259 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:32,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:32,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:32,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:32,259 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,260 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:32,260 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,260 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:32,260 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,260 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,360 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:32,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:32,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:32,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:32,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:32,363 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:32,363 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:32,363 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:32,363 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:32,363 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,364 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,364 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:32,364 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:32,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:32,364 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,365 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:32,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:32,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,365 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:32,365 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:32,365 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:32,365 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:32,365 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:32,365 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:32,365 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:32,366 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:32,366 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:32,366 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:32,366 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:32,366 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:32,366 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:32,366 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:32,366 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,366 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:32,366 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:32,367 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,367 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:32,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:32,368 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:32,368 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:32,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:32,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:32,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:32,369 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:32,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4357a19d[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:32,369 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,369 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,370 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:32,370 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,370 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:32,370 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:32,370 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:32,370 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,370 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:32,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:32,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:32,370 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:32,371 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:32,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:32,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:32,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:32,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:32,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:32,373 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:32,373 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:32,373 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:32,373 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:32,373 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:32,374 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,374 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:32,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:32,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,374 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:32,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:32,374 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:32,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:32,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:32,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,375 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:32,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,376 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,376 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,376 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:32,376 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:32,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:32,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:32,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:32,377 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:32,377 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,377 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,377 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,377 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:32,376 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:32,377 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:32,377 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:32,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:32,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:32,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:32,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987452409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,477 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:32,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:32,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:32,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:32,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:32,480 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:32,480 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:32,480 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:32,480 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:32,480 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,480 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,481 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:32,481 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:32,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:32,481 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,481 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:32,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:32,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,482 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:32,482 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:32,482 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:32,482 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:32,482 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:32,482 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:32,483 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:32,483 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:32,483 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:32,483 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:32,483 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:32,483 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:32,483 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:32,483 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:32,483 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,483 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:32,483 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:32,484 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,484 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:32,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:32,484 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,484 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:32,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:32,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:32,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:32,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:32,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2866ebfb[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:32,485 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:32,485 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,486 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,486 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:32,486 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:32,486 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:32,486 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,486 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:32,487 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,487 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:32,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:32,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:32,487 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:32,487 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:32,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:32,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:32,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:32,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:32,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:32,489 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:32,489 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:32,489 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:32,489 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:32,489 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:32,490 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,490 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:32,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:32,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,490 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:32,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:32,490 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:32,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:32,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:32,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,491 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:32,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,493 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,493 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,493 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,493 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:32,493 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:32,493 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,493 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:32,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:32,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:32,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:32,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:32,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:32,493 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,493 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,493 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,493 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:32,493 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:32,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:32,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:32,495 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=342 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/d42738549e414fd0917ea5bdd5c58e23 2023-04-08 20:56:32,593 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:32,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:32,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:32,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:32,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:32,596 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:32,596 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:32,596 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:32,596 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:32,597 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,597 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,598 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,598 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:32,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:32,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:32,598 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,598 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:32,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:32,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,598 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:32,599 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:32,599 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:32,599 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:32,599 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:32,599 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:32,599 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:32,599 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:32,599 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:32,599 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:32,599 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:32,599 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:32,599 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:32,599 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:32,599 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,599 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:32,599 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:32,600 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,600 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:32,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:32,600 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,600 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:32,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:32,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:32,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:32,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:32,601 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:32,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37aa01ea[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:32,601 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,602 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,602 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:32,602 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:32,602 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:32,602 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,602 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:32,603 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:32,603 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,603 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:32,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:32,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:32,603 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:32,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:32,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:32,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:32,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:32,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:32,604 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:32,604 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:32,604 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:32,604 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:32,604 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:32,605 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,605 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,605 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:32,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:32,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:32,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:32,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:32,605 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:32,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:32,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,606 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:32,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,608 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,608 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:32,608 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,608 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:32,608 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:32,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:32,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:32,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:32,608 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:32,608 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:32,608 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:32,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:32,608 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,608 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:32,608 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,608 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,708 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:32,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:32,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:32,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:32,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:32,711 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:32,711 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:32,711 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:32,711 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:32,712 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,712 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,712 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:32,713 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:32,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:32,713 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,713 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:32,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:32,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,713 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:32,714 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:32,714 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:32,714 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:32,714 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:32,714 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:32,714 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:32,714 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:32,714 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:32,714 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:32,714 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:32,714 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:32,714 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:32,714 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,714 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:32,714 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:32,715 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:32,715 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987452714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:32,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:32,715 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:32,716 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,716 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:32,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:32,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:32,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:32,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52c19a7c[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:32,717 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:32,717 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,717 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,717 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:32,717 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:32,717 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,717 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:32,717 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:32,718 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,718 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:32,718 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:32,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:32,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:32,718 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:32,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:32,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:32,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:32,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:32,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:32,720 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:32,720 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:32,720 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:32,720 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:32,720 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:32,721 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:32,721 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,721 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:32,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:32,722 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:32,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:32,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:32,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,723 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:32,724 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:32,724 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:32,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:32,724 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:32,724 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:32,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:32,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:32,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:32,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:32,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:32,724 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,724 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:32,724 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,725 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:32,725 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,725 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,725 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,725 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,825 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:32,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:32,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:32,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:32,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:32,827 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:32,827 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:32,827 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:32,827 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:32,828 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,828 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,828 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:32,828 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:32,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:32,828 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,829 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:32,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:32,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:32,829 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:32,829 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:32,829 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:32,829 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:32,829 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:32,829 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:32,829 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:32,829 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:32,830 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:32,830 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:32,830 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:32,830 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:32,830 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:32,830 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,830 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:32,830 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:32,831 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,831 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:32,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:32,831 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,831 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:32,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:32,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:32,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:32,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:32,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c6482f0[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:32,832 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:32,832 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,833 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,833 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:32,833 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:32,833 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:32,833 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,833 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:32,834 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,834 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:32,834 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:32,834 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:32,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:32,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:32,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:32,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:32,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:32,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:32,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:32,836 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:32,836 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:32,836 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:32,836 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:32,836 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:32,836 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,836 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,836 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:32,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:32,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:32,837 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:32,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,838 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:32,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,839 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,839 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:32,839 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:32,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:32,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:32,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:32,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:32,839 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:32,839 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:32,839 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:32,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:32,840 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:32,840 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:32,840 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,840 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,840 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,840 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,903 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=342 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/b4b76e865899497d8c93643246097850 2023-04-08 20:56:32,940 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:32,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:32,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:32,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:32,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:32,943 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:32,943 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:32,943 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:32,943 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:32,944 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,944 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,944 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,944 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:32,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:32,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:32,944 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,944 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:32,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:32,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,945 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:32,945 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:32,945 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:32,945 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:32,945 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:32,945 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:32,946 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:32,946 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:32,946 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:32,946 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:32,946 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:32,946 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:32,946 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:32,946 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:32,946 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,946 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:32,946 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:32,947 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,947 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:32,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:32,947 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,947 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:32,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:32,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:32,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:32,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:32,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33429db4[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:32,948 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:32,948 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,949 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,949 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,950 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:32,949 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:32,950 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:32,950 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:32,950 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,950 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:32,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:32,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:32,950 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:32,950 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:32,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:32,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:32,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:32,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:32,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:32,952 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:32,953 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:32,953 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:32,953 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:32,953 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:32,953 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,953 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,953 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:32,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:32,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:32,954 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:32,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:32,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:32,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:32,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:32,955 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:32,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:32,956 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,956 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:32,956 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:32,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:32,956 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:32,956 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:32,956 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:32,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:32,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:32,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:32,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:32,956 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:32,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:32,956 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:32,956 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,956 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:32,957 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:32,957 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:33,057 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:33,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:33,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:33,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:33,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:33,059 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:33,059 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:33,059 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:33,059 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:33,060 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:33,060 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:33,061 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:33,061 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:33,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:33,061 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,061 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:33,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:33,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:33,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:33,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:33,062 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:33,062 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:33,062 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:33,062 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:33,062 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:33,062 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:33,062 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:33,062 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:33,062 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:33,062 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:33,063 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:33,063 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:33,063 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:33,063 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:33,063 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,063 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:33,063 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:33,063 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,063 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:33,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:33,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:33,064 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:33,064 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:33,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:33,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:33,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:33,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:33,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:33,065 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:33,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54807f67[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:33,065 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:33,065 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:33,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:33,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:33,066 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:33,066 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,066 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:33,066 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:33,066 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:33,066 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:33,066 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:33,066 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:33,067 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:33,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:33,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:33,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:33,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:33,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:33,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:33,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:33,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:33,069 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:33,069 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:33,069 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:33,069 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:33,069 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:33,069 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:33,070 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:33,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:33,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:33,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:33,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:33,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:33,070 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:33,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:33,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:33,070 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:33,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:33,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:33,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:33,072 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,073 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,073 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:33,073 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:33,073 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,073 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:33,073 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:33,073 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:33,073 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:33,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:33,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:33,074 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:33,074 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:33,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:33,078 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:33,078 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:33,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:33,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:33,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:33,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:33,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:33,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:33,174 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:33,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:33,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:33,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:33,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:33,176 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:33,177 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:33,177 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:33,177 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:33,177 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:33,177 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:33,178 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,178 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:33,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:33,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:33,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:33,178 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,178 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:33,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:33,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:33,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:33,179 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:33,179 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:33,179 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:33,179 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:33,179 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:33,179 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:33,179 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:33,179 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:33,179 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:33,179 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:33,179 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:33,179 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:33,179 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:33,180 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:33,180 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,180 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:33,180 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:33,180 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:33,180 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:33,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:33,181 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:33,181 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:33,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:33,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:33,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:33,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:33,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:33,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@acc0df1[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:33,182 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:33,182 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:33,182 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:33,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:33,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:33,183 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:33,183 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:33,183 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:33,183 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,183 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:33,183 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,183 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:33,183 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:33,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:33,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:33,183 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:33,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:33,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:33,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:33,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:33,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:33,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:33,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:33,185 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:33,185 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:33,185 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:33,185 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:33,185 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:33,186 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:33,186 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:33,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:33,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:33,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:33,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:33,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:33,186 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:33,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:33,186 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:33,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:33,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:33,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:33,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:33,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:33,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,188 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:33,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:33,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,189 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,189 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:33,189 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:33,189 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:33,189 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:33,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:33,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:33,189 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:33,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:33,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:33,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:33,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:33,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:33,189 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,189 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:33,189 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:33,189 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:33,190 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:33,190 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:33,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:33,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987453216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,290 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:33,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:33,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:33,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:33,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:33,292 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:33,292 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:33,292 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:33,292 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:33,293 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:33,293 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:33,293 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,293 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:33,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:33,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:33,293 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,293 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:33,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:33,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:33,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:33,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:33,294 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:33,296 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:33,296 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:33,296 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:33,296 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:33,296 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:33,297 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:33,297 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:33,297 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:33,297 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:33,297 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:33,297 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:33,297 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:33,297 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:33,297 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,297 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:33,297 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:33,298 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,298 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:33,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:33,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:33,298 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:33,298 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:33,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:33,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:33,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:33,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:33,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:33,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2376d5a[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:33,299 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:33,299 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:33,300 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:33,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:33,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:33,300 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,300 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:33,300 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:33,300 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:33,300 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:33,301 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,301 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:33,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:33,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:33,302 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:33,302 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:33,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:33,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:33,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:33,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:33,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:33,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:33,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:33,303 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:33,303 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:33,303 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:33,303 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:33,303 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:33,304 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:33,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:33,304 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:33,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:33,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:33,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:33,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:33,304 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:33,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:33,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:33,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:33,305 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:33,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:33,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:33,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:33,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,306 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:33,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:33,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,307 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:33,307 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:33,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:33,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:33,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:33,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:33,308 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,308 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:33,308 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:33,309 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,309 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:33,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:33,309 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:33,309 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:33,309 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:33,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:33,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:33,309 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:33,309 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:33,313 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=342 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/9d90d95b0a2f46198af639d009596b63 2023-04-08 20:56:33,317 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/d42738549e414fd0917ea5bdd5c58e23 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/d42738549e414fd0917ea5bdd5c58e23 2023-04-08 20:56:33,320 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/d42738549e414fd0917ea5bdd5c58e23, entries=200, sequenceid=342, filesize=14.3 K 2023-04-08 20:56:33,321 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/b4b76e865899497d8c93643246097850 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/b4b76e865899497d8c93643246097850 2023-04-08 20:56:33,325 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/b4b76e865899497d8c93643246097850, entries=150, sequenceid=342, filesize=11.9 K 2023-04-08 20:56:33,326 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/9d90d95b0a2f46198af639d009596b63 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/9d90d95b0a2f46198af639d009596b63 2023-04-08 20:56:33,330 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/9d90d95b0a2f46198af639d009596b63, entries=150, sequenceid=342, filesize=11.9 K 2023-04-08 20:56:33,330 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for ebdeb8625c2b4ac57e00f30df2ca3028 in 1252ms, sequenceid=342, compaction requested=true 2023-04-08 20:56:33,330 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:33,331 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:56:33,331 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-08 20:56:33,331 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-08 20:56:33,331 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:56:33,331 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:56:33,332 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61506 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-08 20:56:33,333 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59066 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-08 20:56:33,333 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): ebdeb8625c2b4ac57e00f30df2ca3028/A is initiating minor compaction (all files) 2023-04-08 20:56:33,333 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): ebdeb8625c2b4ac57e00f30df2ca3028/B is initiating minor compaction (all files) 2023-04-08 20:56:33,333 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ebdeb8625c2b4ac57e00f30df2ca3028/A in TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:33,333 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ebdeb8625c2b4ac57e00f30df2ca3028/B in TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:33,333 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/83e2f2aa1ecd4324b9465a8a099b845a, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/788218e7680b42c98dd2c52211606911, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/cda2ad3a1fb74bb7b9577e75585ddcb9, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/3cf5514e5c074f3ebbba51ddf7513a54, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/d42738549e414fd0917ea5bdd5c58e23] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp, totalSize=60.1 K 2023-04-08 20:56:33,333 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/00f2634921d8410fb9224df22bf2bfbb, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/e3990c72148940b9aa72053582507e8b, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/1e88adf60591498697b0a402312bec89, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/b10b0b4ec47f4669ae501870e28dc848, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/b4b76e865899497d8c93643246097850] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp, totalSize=57.7 K 2023-04-08 20:56:33,333 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 83e2f2aa1ecd4324b9465a8a099b845a, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=274, earliestPutTs=1680987387355 2023-04-08 20:56:33,333 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 00f2634921d8410fb9224df22bf2bfbb, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=274, earliestPutTs=1680987387355 2023-04-08 20:56:33,334 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 788218e7680b42c98dd2c52211606911, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1680987387991 2023-04-08 20:56:33,334 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting e3990c72148940b9aa72053582507e8b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1680987387991 2023-04-08 20:56:33,334 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting cda2ad3a1fb74bb7b9577e75585ddcb9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1680987389133 2023-04-08 20:56:33,334 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 1e88adf60591498697b0a402312bec89, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1680987389133 2023-04-08 20:56:33,334 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 3cf5514e5c074f3ebbba51ddf7513a54, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1680987390287 2023-04-08 20:56:33,338 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting b10b0b4ec47f4669ae501870e28dc848, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1680987390287 2023-04-08 20:56:33,338 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting d42738549e414fd0917ea5bdd5c58e23, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1680987390951 2023-04-08 20:56:33,339 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting b4b76e865899497d8c93643246097850, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1680987390951 2023-04-08 20:56:33,348 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): ebdeb8625c2b4ac57e00f30df2ca3028#A#compaction#277 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:56:33,350 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): ebdeb8625c2b4ac57e00f30df2ca3028#B#compaction#278 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:56:33,409 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:33,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:33,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:33,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:33,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:33,412 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:33,412 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:33,412 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:33,412 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:33,412 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:33,412 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:33,413 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:33,413 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:33,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:33,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:33,414 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:33,414 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:33,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:33,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:33,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:33,415 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:33,415 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:33,415 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:33,415 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:33,415 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:33,415 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:33,415 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:33,415 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:33,415 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:33,415 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:33,416 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ebdeb8625c2b4ac57e00f30df2ca3028 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-08 20:56:33,416 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=A 2023-04-08 20:56:33,416 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:33,416 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=B 2023-04-08 20:56:33,416 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:33,416 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=C 2023-04-08 20:56:33,416 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:33,771 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/eaeb7153c74b42b887c30c0f90b4d185 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/eaeb7153c74b42b887c30c0f90b4d185 2023-04-08 20:56:33,771 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/bd74065e3eb94d3b92ccc230b0a5dd9d as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/bd74065e3eb94d3b92ccc230b0a5dd9d 2023-04-08 20:56:33,775 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ebdeb8625c2b4ac57e00f30df2ca3028/B of ebdeb8625c2b4ac57e00f30df2ca3028 into bd74065e3eb94d3b92ccc230b0a5dd9d(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:56:33,775 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ebdeb8625c2b4ac57e00f30df2ca3028/A of ebdeb8625c2b4ac57e00f30df2ca3028 into eaeb7153c74b42b887c30c0f90b4d185(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:56:33,775 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:33,775 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:33,775 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028., storeName=ebdeb8625c2b4ac57e00f30df2ca3028/B, priority=11, startTime=1680987393331; duration=0sec 2023-04-08 20:56:33,775 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028., storeName=ebdeb8625c2b4ac57e00f30df2ca3028/A, priority=11, startTime=1680987393331; duration=0sec 2023-04-08 20:56:33,775 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:56:33,775 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:56:33,775 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-08 20:56:33,776 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58998 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-08 20:56:33,777 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): ebdeb8625c2b4ac57e00f30df2ca3028/C is initiating minor compaction (all files) 2023-04-08 20:56:33,777 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ebdeb8625c2b4ac57e00f30df2ca3028/C in TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:33,777 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/c6aadbd82cb54dc4b86ab68af81766e5, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/9327d46b760749cf82d7fed39793361b, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/ecb18c00d31c4b1a839eaf791be7ebfe, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/5a64000fc6874832ba507f46d4d2f715, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/9d90d95b0a2f46198af639d009596b63] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp, totalSize=57.6 K 2023-04-08 20:56:33,777 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting c6aadbd82cb54dc4b86ab68af81766e5, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=274, earliestPutTs=1680987387355 2023-04-08 20:56:33,777 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 9327d46b760749cf82d7fed39793361b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1680987387991 2023-04-08 20:56:33,778 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting ecb18c00d31c4b1a839eaf791be7ebfe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1680987389133 2023-04-08 20:56:33,778 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 5a64000fc6874832ba507f46d4d2f715, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1680987390287 2023-04-08 20:56:33,778 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 9d90d95b0a2f46198af639d009596b63, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1680987390951 2023-04-08 20:56:33,803 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): ebdeb8625c2b4ac57e00f30df2ca3028#C#compaction#280 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:56:33,811 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/68884a2f8eb94220a0700d674276fcb3 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/68884a2f8eb94220a0700d674276fcb3 2023-04-08 20:56:33,814 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ebdeb8625c2b4ac57e00f30df2ca3028/C of ebdeb8625c2b4ac57e00f30df2ca3028 into 68884a2f8eb94220a0700d674276fcb3(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:56:33,815 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:33,815 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028., storeName=ebdeb8625c2b4ac57e00f30df2ca3028/C, priority=11, startTime=1680987393331; duration=0sec 2023-04-08 20:56:33,815 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:56:33,843 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=353 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/e3ad4eeebce84388813f48dfdb0f1af2 2023-04-08 20:56:34,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on ebdeb8625c2b4ac57e00f30df2ca3028 2023-04-08 20:56:34,229 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:34,252 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=353 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/1612c51892c0456ea5d07002479f1bea 2023-04-08 20:56:34,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680987394261, max timestamp=1680987394269Num uniques 859; 2023-04-08 20:56:34,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 20:56:34,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680987394261, max timestamp=1680987394269Num uniques 859; 2023-04-08 20:56:34,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 20:56:34,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680987394261, max timestamp=1680987394269Num uniques 859; 2023-04-08 20:56:34,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 20:56:34,311 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:34,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987454309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:34,413 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:34,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987454412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:34,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:34,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987454614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:34,677 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=353 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/eeebdf7078e74be5b9dbb25b2bac4fb4 2023-04-08 20:56:34,681 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/e3ad4eeebce84388813f48dfdb0f1af2 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/e3ad4eeebce84388813f48dfdb0f1af2 2023-04-08 20:56:34,686 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/e3ad4eeebce84388813f48dfdb0f1af2, entries=150, sequenceid=353, filesize=11.9 K 2023-04-08 20:56:34,686 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/1612c51892c0456ea5d07002479f1bea as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/1612c51892c0456ea5d07002479f1bea 2023-04-08 20:56:34,689 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/1612c51892c0456ea5d07002479f1bea, entries=150, sequenceid=353, filesize=11.9 K 2023-04-08 20:56:34,690 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/eeebdf7078e74be5b9dbb25b2bac4fb4 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/eeebdf7078e74be5b9dbb25b2bac4fb4 2023-04-08 20:56:34,695 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/eeebdf7078e74be5b9dbb25b2bac4fb4, entries=150, sequenceid=353, filesize=11.9 K 2023-04-08 20:56:34,696 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for ebdeb8625c2b4ac57e00f30df2ca3028 in 1280ms, sequenceid=353, compaction requested=false 2023-04-08 20:56:34,696 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:34,696 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:34,696 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:34,696 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:34,696 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:34,696 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:34,696 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:34,697 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:34,697 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:34,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:34,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:34,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:34,697 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:34,697 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:34,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:34,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:34,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:34,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:34,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:34,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:34,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@253fff95[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:34,698 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:34,698 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:34,699 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:34,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:34,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:34,699 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:34,699 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:34,699 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:34,699 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:34,699 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:34,700 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:34,700 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:34,700 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:34,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:34,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:34,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:34,700 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:34,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:34,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:34,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:34,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:34,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:34,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:34,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:34,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:34,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:34,701 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:34,701 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:34,701 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:34,702 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:34,702 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:34,702 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:34,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:34,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:34,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:34,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:34,703 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:34,703 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:34,703 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:34,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:34,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:34,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:34,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:34,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:34,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:34,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:34,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:34,704 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:34,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:34,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:34,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:34,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:34,705 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:34,705 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:34,705 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:34,705 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:34,705 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:34,705 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:34,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:34,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:34,705 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:34,705 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:34,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:34,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:34,705 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:34,705 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:34,706 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:34,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:34,706 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:34,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:34,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:34,806 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:34,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:34,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:34,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:34,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:34,809 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:34,809 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:34,809 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:34,809 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:34,809 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:34,809 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:34,810 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:34,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:34,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:34,810 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:34,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:34,810 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:34,810 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:34,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:34,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:34,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:34,810 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:34,811 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:34,811 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:34,811 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:34,811 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:34,811 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:34,811 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:34,811 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:34,811 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:34,811 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:34,811 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ebdeb8625c2b4ac57e00f30df2ca3028 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-08 20:56:34,811 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=A 2023-04-08 20:56:34,811 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:56:34,812 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=B 2023-04-08 20:56:34,812 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:56:34,812 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=C 2023-04-08 20:56:34,812 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:56:34,818 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=392 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/31e755da37364f0da7729b756347925b 2023-04-08 20:56:34,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on ebdeb8625c2b4ac57e00f30df2ca3028 2023-04-08 20:56:34,919 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:34,947 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:34,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 363 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987454944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:35,049 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:35,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 365 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987455048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:35,227 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=392 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/6ac6acc784e742af8d2bf43e1a587e8b 2023-04-08 20:56:35,252 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:35,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 367 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987455251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:35,556 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:35,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 369 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987455555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:35,636 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=392 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/ad281e7e07584c5a9d4aef4c61e2f0be 2023-04-08 20:56:35,640 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/31e755da37364f0da7729b756347925b as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/31e755da37364f0da7729b756347925b 2023-04-08 20:56:35,643 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/31e755da37364f0da7729b756347925b, entries=150, sequenceid=392, filesize=11.9 K 2023-04-08 20:56:35,644 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/6ac6acc784e742af8d2bf43e1a587e8b as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/6ac6acc784e742af8d2bf43e1a587e8b 2023-04-08 20:56:35,647 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/6ac6acc784e742af8d2bf43e1a587e8b, entries=150, sequenceid=392, filesize=11.9 K 2023-04-08 20:56:35,648 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/ad281e7e07584c5a9d4aef4c61e2f0be as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/ad281e7e07584c5a9d4aef4c61e2f0be 2023-04-08 20:56:35,653 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/ad281e7e07584c5a9d4aef4c61e2f0be, entries=150, sequenceid=392, filesize=11.9 K 2023-04-08 20:56:35,653 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for ebdeb8625c2b4ac57e00f30df2ca3028 in 842ms, sequenceid=392, compaction requested=true 2023-04-08 20:56:35,654 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:35,654 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:35,654 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:35,654 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:35,654 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:35,654 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:35,654 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:35,655 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:35,655 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:35,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:35,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:35,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:35,655 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:35,655 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:35,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:35,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:35,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:35,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:35,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:35,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:35,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7293d712[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:35,657 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:35,657 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:35,658 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:35,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:35,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:35,658 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:35,658 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:35,659 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:35,659 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:35,659 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:35,659 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:35,659 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:35,659 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:35,659 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:35,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:35,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:35,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:35,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:35,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:35,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:35,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:35,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:35,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:35,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:35,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:35,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:35,662 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:35,662 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:35,662 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:35,662 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:35,662 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:35,663 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:35,663 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:35,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:35,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:35,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:35,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:35,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:35,663 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:35,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:35,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:35,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:35,664 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:35,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:35,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:35,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:35,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:35,665 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:35,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:35,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:35,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:35,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:35,666 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:35,666 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:35,666 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:35,666 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:35,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:35,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:35,667 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:35,667 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:35,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:35,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:35,667 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:35,667 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:35,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:35,666 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:35,667 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:35,667 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:35,667 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:35,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:35,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:35,767 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:35,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:35,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:35,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:35,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:35,771 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:35,771 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:35,771 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:35,771 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:35,771 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:35,771 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:35,772 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:35,772 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:35,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:35,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:35,772 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:35,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:35,772 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:35,772 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:35,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:35,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:35,773 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:35,773 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:35,773 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:35,773 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:35,773 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:35,773 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:35,773 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:35,773 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:35,773 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:35,773 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:35,774 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ebdeb8625c2b4ac57e00f30df2ca3028 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-08 20:56:35,774 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=A 2023-04-08 20:56:35,774 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:35,774 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=B 2023-04-08 20:56:35,774 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:35,774 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=C 2023-04-08 20:56:35,774 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:35,780 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=403 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/ce1b4d93432a487f9d60a87063ee8629 2023-04-08 20:56:36,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on ebdeb8625c2b4ac57e00f30df2ca3028 2023-04-08 20:56:36,060 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:36,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680987396085, max timestamp=1680987396105Num uniques 859; 2023-04-08 20:56:36,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 20:56:36,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680987396085, max timestamp=1680987396105Num uniques 859; 2023-04-08 20:56:36,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 20:56:36,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680987396085, max timestamp=1680987396105Num uniques 859; 2023-04-08 20:56:36,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 20:56:36,157 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:36,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 404 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987456154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,216 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=403 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/aebc8b92324f48ccac3dbeafe0a517d1 2023-04-08 20:56:36,224 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=403 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/5e0c9912a89f4207ac6db281d00f9e40 2023-04-08 20:56:36,227 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/ce1b4d93432a487f9d60a87063ee8629 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/ce1b4d93432a487f9d60a87063ee8629 2023-04-08 20:56:36,231 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/ce1b4d93432a487f9d60a87063ee8629, entries=150, sequenceid=403, filesize=11.9 K 2023-04-08 20:56:36,232 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/aebc8b92324f48ccac3dbeafe0a517d1 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/aebc8b92324f48ccac3dbeafe0a517d1 2023-04-08 20:56:36,235 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/aebc8b92324f48ccac3dbeafe0a517d1, entries=150, sequenceid=403, filesize=11.9 K 2023-04-08 20:56:36,236 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/5e0c9912a89f4207ac6db281d00f9e40 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/5e0c9912a89f4207ac6db281d00f9e40 2023-04-08 20:56:36,239 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/5e0c9912a89f4207ac6db281d00f9e40, entries=150, sequenceid=403, filesize=11.9 K 2023-04-08 20:56:36,240 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for ebdeb8625c2b4ac57e00f30df2ca3028 in 466ms, sequenceid=403, compaction requested=true 2023-04-08 20:56:36,240 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:36,240 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:36,240 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:36,240 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:36,240 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,240 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:36,240 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:36,241 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,241 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:36,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:36,241 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,241 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:36,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:36,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:36,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:36,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:36,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18daff34[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:36,243 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:36,243 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,243 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,243 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:36,243 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:36,243 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:36,243 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,243 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:36,244 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,244 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:36,244 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:36,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:36,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:36,244 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:36,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:36,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:36,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:36,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:36,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:36,246 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:36,246 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:36,246 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:36,246 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:36,246 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:36,247 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,247 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:36,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:36,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,247 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:36,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,248 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:36,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:36,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:36,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:36,248 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:36,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,250 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:36,250 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:36,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:36,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:36,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:36,250 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:36,250 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:36,250 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:36,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:36,250 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:36,250 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:36,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:36,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:36,251 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:36,251 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,251 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,251 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,251 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on ebdeb8625c2b4ac57e00f30df2ca3028 2023-04-08 20:56:36,262 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ebdeb8625c2b4ac57e00f30df2ca3028 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-08 20:56:36,262 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=A 2023-04-08 20:56:36,262 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:56:36,262 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=B 2023-04-08 20:56:36,262 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:56:36,262 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=C 2023-04-08 20:56:36,262 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:56:36,293 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:36,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 414 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987456289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,351 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:36,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:36,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:36,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:36,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:36,353 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:36,353 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:36,354 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:36,354 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:36,354 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,354 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:36,355 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,355 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:36,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:36,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:36,355 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,355 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:36,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:36,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:36,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:36,356 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:36,356 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:36,356 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:36,356 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:36,356 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:36,356 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:36,356 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:36,356 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:36,356 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:36,356 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:36,356 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:36,356 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:36,356 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:36,356 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:36,356 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,356 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:36,356 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:36,357 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,357 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:36,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:36,357 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,357 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:36,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:36,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:36,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:36,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:36,359 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:36,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@213e0cc8[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:36,359 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,359 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,359 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:36,359 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:36,359 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:36,359 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,359 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:36,360 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,360 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:36,360 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:36,360 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:36,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:36,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:36,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:36,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:36,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:36,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:36,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:36,362 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:36,363 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:36,363 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:36,363 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:36,363 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:36,364 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,364 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:36,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:36,364 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:36,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:36,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:36,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:36,364 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:36,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,365 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:36,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,367 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,367 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:36,367 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:36,367 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:36,367 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:36,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:36,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:36,367 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,367 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:36,367 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:36,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:36,367 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:36,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:36,367 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:36,367 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,367 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:36,396 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:36,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 416 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987456394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,468 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:36,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:36,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:36,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:36,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:36,471 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:36,471 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:36,471 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:36,471 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:36,471 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,471 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:36,472 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:36,472 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:36,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:36,472 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,472 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:36,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:36,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:36,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:36,473 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:36,475 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:36,475 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:36,475 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:36,475 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:36,475 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:36,475 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:36,475 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:36,475 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:36,475 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:36,476 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:36,476 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:36,476 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:36,476 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:36,476 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,476 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:36,476 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:36,476 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,477 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:36,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:36,477 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,477 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:36,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:36,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:36,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:36,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:36,478 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:36,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ba79c2d[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:36,478 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,479 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,479 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:36,479 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:36,479 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:36,479 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,479 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:36,479 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,479 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:36,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:36,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:36,479 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:36,480 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:36,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:36,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:36,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:36,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:36,482 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:36,482 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:36,482 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:36,482 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:36,482 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:36,482 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,482 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:36,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:36,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,482 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:36,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:36,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:36,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:36,483 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:36,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,484 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:36,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,485 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,485 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:36,485 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:36,485 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:36,485 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:36,485 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:36,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:36,485 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,485 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,485 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,485 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:36,485 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:36,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:36,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:36,485 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:36,486 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:36,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:36,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:36,586 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:36,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:36,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:36,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:36,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:36,589 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:36,589 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:36,589 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:36,589 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:36,589 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,589 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:36,590 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,590 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:36,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:36,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:36,590 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,590 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:36,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:36,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:36,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:36,591 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:36,591 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:36,591 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:36,591 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:36,591 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:36,591 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:36,591 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:36,591 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:36,591 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:36,591 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:36,591 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:36,592 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:36,592 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:36,592 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:36,592 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,592 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:36,592 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:36,592 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,592 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:36,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:36,593 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,593 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:36,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:36,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:36,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:36,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:36,594 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:36,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1760ff0c[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:36,594 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,595 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,595 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:36,595 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,595 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:36,595 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:36,595 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:36,595 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,596 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:36,596 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:36,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:36,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:36,596 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:36,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:36,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:36,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:36,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:36,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:36,597 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:36,597 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:36,597 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:36,598 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:36,598 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:36,598 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,598 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:36,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:36,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,599 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:36,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:36,599 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:36,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:36,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:36,599 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:36,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 418 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987456598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,600 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:36,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,602 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,602 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:36,602 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:36,602 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:36,602 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:36,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:36,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:36,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:36,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:36,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:36,602 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:36,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:36,602 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:36,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:36,602 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,602 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:36,603 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,603 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,603 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,668 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=440 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/3a059bf715c24a5589dadc6fcea9aaaf 2023-04-08 20:56:36,703 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:36,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:36,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:36,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:36,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:36,706 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:36,706 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:36,706 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:36,706 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:36,706 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,706 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:36,707 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:36,707 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:36,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:36,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:36,707 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,707 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:36,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:36,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:36,708 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:36,708 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:36,708 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:36,708 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:36,709 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:36,709 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:36,709 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:36,709 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:36,709 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:36,709 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:36,709 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:36,709 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:36,709 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:36,709 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:36,709 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,709 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:36,709 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:36,710 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,710 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:36,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:36,710 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,710 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:36,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:36,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:36,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:36,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:36,711 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:36,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@580c918b[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:36,711 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,712 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,712 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:36,712 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:36,712 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:36,712 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,712 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:36,713 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,713 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:36,713 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:36,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:36,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:36,713 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:36,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:36,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:36,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:36,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:36,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:36,715 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:36,715 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:36,715 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:36,715 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:36,715 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:36,716 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,716 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:36,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:36,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,716 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:36,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:36,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:36,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:36,716 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:36,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,717 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:36,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,719 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:36,719 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,719 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:36,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:36,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:36,719 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:36,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:36,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:36,719 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:36,719 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:36,719 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,719 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,719 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,719 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,719 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:36,719 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:36,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:36,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:36,820 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:36,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:36,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:36,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:36,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:36,822 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:36,822 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:36,822 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:36,822 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:36,822 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,822 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:36,823 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:36,823 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:36,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:36,823 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,823 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:36,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:36,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:36,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:36,824 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:36,824 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:36,824 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:36,824 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:36,824 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:36,824 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:36,825 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:36,825 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:36,825 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:36,825 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:36,825 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:36,825 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:36,825 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:36,825 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:36,825 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,825 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:36,825 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:36,826 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,826 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:36,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:36,826 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,826 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:36,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:36,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:36,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:36,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:36,827 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:36,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9859e0e[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:36,827 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,828 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,828 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:36,828 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:36,828 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:36,828 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,828 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:36,829 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,829 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:36,829 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:36,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:36,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:36,829 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:36,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:36,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:36,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:36,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:36,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:36,830 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:36,830 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:36,830 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:36,830 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:36,831 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:36,831 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,831 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:36,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:36,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,831 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:36,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:36,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:36,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:36,832 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:36,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,833 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:36,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,834 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,834 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:36,834 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:36,834 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:36,834 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:36,834 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:36,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:36,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:36,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:36,834 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:36,834 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,835 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:36,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:36,835 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:36,835 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,835 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:36,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:36,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 420 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987456900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,935 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:36,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:36,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:36,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:36,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:36,937 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:36,937 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:36,937 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:36,937 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:36,938 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,938 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:36,938 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:36,938 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:36,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:36,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:36,939 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,939 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:36,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:36,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:36,939 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:36,940 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:36,940 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:36,940 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:36,940 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:36,940 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:36,940 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:36,940 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:36,940 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:36,940 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:36,940 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:36,940 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:36,940 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:36,940 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:36,940 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,940 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:36,940 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:36,941 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:36,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:36,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:36,941 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:36,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:36,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:36,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e842de7[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:36,942 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:36,942 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,942 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,942 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:36,943 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,943 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:36,943 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,943 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:36,943 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:36,943 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:36,944 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:36,944 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:36,944 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,944 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:36,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:36,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:36,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:36,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:36,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:36,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:36,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:36,946 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:36,946 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:36,946 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:36,946 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:36,946 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:36,946 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,946 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:36,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:36,946 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:36,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:36,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:36,947 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:36,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:36,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,948 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:36,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:36,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,949 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,949 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:36,949 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:36,949 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:36,949 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,949 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:36,949 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:36,949 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:36,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:36,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:36,949 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:36,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:36,949 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:36,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:36,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:36,949 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:36,949 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:36,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:36,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:37,049 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:37,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:37,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:37,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:37,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:37,052 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:37,052 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:37,052 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:37,052 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:37,053 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:37,053 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:37,054 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:37,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:37,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:37,054 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:37,054 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,054 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:37,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:37,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:37,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:37,055 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:37,055 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:37,055 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:37,055 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:37,055 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:37,055 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:37,055 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:37,055 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:37,055 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:37,055 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:37,055 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:37,055 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:37,055 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:37,056 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:37,056 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,056 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:37,056 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:37,056 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,056 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:37,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:37,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:37,057 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:37,057 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:37,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:37,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:37,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:37,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:37,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:37,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e8c23a4[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:37,058 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:37,058 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:37,059 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:37,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:37,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:37,059 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:37,059 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:37,059 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:37,059 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,059 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:37,060 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,060 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:37,060 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:37,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:37,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:37,060 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:37,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:37,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:37,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:37,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:37,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:37,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:37,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:37,062 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:37,062 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:37,062 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:37,062 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:37,062 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:37,064 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:37,064 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:37,064 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:37,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:37,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:37,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:37,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:37,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:37,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:37,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:37,064 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:37,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:37,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:37,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:37,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:37,065 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:37,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:37,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,067 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:37,067 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:37,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:37,067 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:37,067 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:37,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:37,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:37,067 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:37,067 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:37,067 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,067 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:37,067 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:37,067 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:37,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:37,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:37,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:37,068 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:37,068 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:37,079 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=440 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/97340d0fe2e54eb684528437f8986a45 2023-04-08 20:56:37,168 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:37,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:37,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:37,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:37,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:37,171 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:37,171 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:37,171 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:37,171 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:37,172 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:37,172 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:37,172 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,172 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:37,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:37,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:37,173 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,173 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:37,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:37,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:37,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:37,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:37,173 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:37,174 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:37,174 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:37,174 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:37,174 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:37,174 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:37,174 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:37,174 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:37,174 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:37,174 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:37,174 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:37,174 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:37,174 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:37,174 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:37,174 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,174 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:37,174 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:37,175 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,175 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:37,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:37,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:37,176 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:37,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:37,176 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:37,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:37,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:37,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:37,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:37,177 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:37,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62b07137[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:37,177 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:37,177 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:37,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:37,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:37,177 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,177 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:37,177 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:37,177 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:37,177 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:37,178 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,178 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:37,178 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:37,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:37,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:37,178 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:37,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:37,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:37,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:37,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:37,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:37,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:37,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:37,180 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:37,180 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:37,180 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:37,180 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:37,180 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:37,181 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:37,181 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:37,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:37,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:37,181 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:37,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:37,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:37,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:37,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:37,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:37,181 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:37,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:37,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:37,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:37,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:37,183 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:37,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:37,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,185 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:37,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:37,185 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:37,185 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:37,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:37,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:37,185 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:37,185 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:37,186 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:37,186 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,186 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:37,186 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:37,186 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:37,186 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:37,186 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:37,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:37,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:37,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:37,286 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:37,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:37,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:37,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:37,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:37,289 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:37,289 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:37,289 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:37,289 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:37,290 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:37,290 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:37,290 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:37,290 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:37,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:37,291 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,291 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:37,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:37,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:37,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:37,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:37,291 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:37,291 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:37,291 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:37,292 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:37,292 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:37,292 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:37,292 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:37,292 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:37,292 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:37,292 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:37,292 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:37,292 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:37,292 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:37,292 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:37,292 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,292 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:37,292 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:37,293 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,293 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:37,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:37,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:37,293 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:37,293 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:37,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:37,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:37,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:37,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:37,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:37,294 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:37,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47e0e681[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:37,294 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:37,295 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:37,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:37,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:37,295 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:37,295 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:37,295 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:37,295 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,295 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:37,296 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,296 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:37,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:37,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:37,296 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:37,296 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:37,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:37,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:37,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:37,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:37,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:37,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:37,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:37,297 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:37,297 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:37,297 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:37,297 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:37,297 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:37,298 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:37,298 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:37,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:37,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:37,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:37,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:37,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:37,298 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:37,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:37,299 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:37,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:37,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:37,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:37,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:37,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:37,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,300 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:37,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:37,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,301 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,301 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:37,301 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:37,301 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:37,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:37,301 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:37,301 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,301 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:37,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:37,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:37,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:37,301 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:37,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:37,301 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:37,301 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:37,301 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:37,302 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:37,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:37,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:37,402 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:37,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:37,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:37,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:37,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:37,405 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:37,405 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:37,405 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:37,405 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:37,405 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:37,405 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:37,406 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,406 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:37,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:37,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:37,406 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,406 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:37,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:37,407 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:37,407 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:37,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 422 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44472 deadline: 1680987457405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:37,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:37,407 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:37,407 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:37,407 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:37,407 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:37,407 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:37,407 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:37,407 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:37,407 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:37,407 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:37,408 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:37,408 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:37,408 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:37,408 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:37,408 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:37,408 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,408 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:37,408 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:37,409 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,409 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:37,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:37,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:37,409 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:37,409 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:37,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:37,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:37,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:37,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:37,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:37,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2864108f[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:37,411 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:37,411 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:37,412 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:37,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:37,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:37,412 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:37,412 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,412 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:37,412 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:37,412 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:37,413 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,413 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:37,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:37,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:37,413 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:37,413 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:37,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:37,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:37,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:37,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:37,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:37,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:37,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:37,415 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:37,415 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:37,415 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:37,415 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:37,415 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:37,416 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:37,416 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:37,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:37,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:37,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:37,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:37,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:37,416 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:37,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:37,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:37,417 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:37,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:37,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:37,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:37,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:37,418 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:37,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:37,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,419 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,419 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:37,419 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:37,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:37,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:37,419 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:37,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:37,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:37,419 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:37,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:37,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:37,419 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:37,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:37,419 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:37,419 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:37,419 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,420 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:37,420 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:37,420 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:37,487 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=440 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/7cd0514dd5f7433e91ed7747d99d91e6 2023-04-08 20:56:37,491 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/3a059bf715c24a5589dadc6fcea9aaaf as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/3a059bf715c24a5589dadc6fcea9aaaf 2023-04-08 20:56:37,494 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/3a059bf715c24a5589dadc6fcea9aaaf, entries=200, sequenceid=440, filesize=14.3 K 2023-04-08 20:56:37,494 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/97340d0fe2e54eb684528437f8986a45 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/97340d0fe2e54eb684528437f8986a45 2023-04-08 20:56:37,499 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/97340d0fe2e54eb684528437f8986a45, entries=150, sequenceid=440, filesize=11.9 K 2023-04-08 20:56:37,500 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/7cd0514dd5f7433e91ed7747d99d91e6 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/7cd0514dd5f7433e91ed7747d99d91e6 2023-04-08 20:56:37,503 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/7cd0514dd5f7433e91ed7747d99d91e6, entries=150, sequenceid=440, filesize=11.9 K 2023-04-08 20:56:37,504 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for ebdeb8625c2b4ac57e00f30df2ca3028 in 1243ms, sequenceid=440, compaction requested=true 2023-04-08 20:56:37,504 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:37,504 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:56:37,504 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-08 20:56:37,504 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-08 20:56:37,504 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:56:37,504 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:56:37,505 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64120 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-08 20:56:37,506 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): ebdeb8625c2b4ac57e00f30df2ca3028/A is initiating minor compaction (all files) 2023-04-08 20:56:37,506 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ebdeb8625c2b4ac57e00f30df2ca3028/A in TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:37,506 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/eaeb7153c74b42b887c30c0f90b4d185, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/e3ad4eeebce84388813f48dfdb0f1af2, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/31e755da37364f0da7729b756347925b, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/ce1b4d93432a487f9d60a87063ee8629, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/3a059bf715c24a5589dadc6fcea9aaaf] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp, totalSize=62.6 K 2023-04-08 20:56:37,506 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61680 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-08 20:56:37,506 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): ebdeb8625c2b4ac57e00f30df2ca3028/B is initiating minor compaction (all files) 2023-04-08 20:56:37,506 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ebdeb8625c2b4ac57e00f30df2ca3028/B in TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:37,506 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/bd74065e3eb94d3b92ccc230b0a5dd9d, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/1612c51892c0456ea5d07002479f1bea, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/6ac6acc784e742af8d2bf43e1a587e8b, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/aebc8b92324f48ccac3dbeafe0a517d1, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/97340d0fe2e54eb684528437f8986a45] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp, totalSize=60.2 K 2023-04-08 20:56:37,507 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting eaeb7153c74b42b887c30c0f90b4d185, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1680987390951 2023-04-08 20:56:37,507 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting bd74065e3eb94d3b92ccc230b0a5dd9d, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1680987390951 2023-04-08 20:56:37,507 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting e3ad4eeebce84388813f48dfdb0f1af2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1680987392091 2023-04-08 20:56:37,507 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 1612c51892c0456ea5d07002479f1bea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1680987392091 2023-04-08 20:56:37,507 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 31e755da37364f0da7729b756347925b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1680987394295 2023-04-08 20:56:37,507 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 6ac6acc784e742af8d2bf43e1a587e8b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1680987394295 2023-04-08 20:56:37,507 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting ce1b4d93432a487f9d60a87063ee8629, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1680987394931 2023-04-08 20:56:37,507 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting aebc8b92324f48ccac3dbeafe0a517d1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1680987394931 2023-04-08 20:56:37,508 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 3a059bf715c24a5589dadc6fcea9aaaf, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=440, earliestPutTs=1680987396141 2023-04-08 20:56:37,508 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 97340d0fe2e54eb684528437f8986a45, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=440, earliestPutTs=1680987396141 2023-04-08 20:56:37,520 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:37,520 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): ebdeb8625c2b4ac57e00f30df2ca3028#B#compaction#292 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:56:37,520 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): ebdeb8625c2b4ac57e00f30df2ca3028#A#compaction#293 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:56:37,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:37,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:37,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:37,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:37,526 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:37,526 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:37,526 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:37,526 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:37,526 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:37,526 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:37,527 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:37,527 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:37,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:37,527 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:37,528 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:37,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:37,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:37,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:37,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:37,528 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:37,534 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:37,534 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:37,534 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:37,534 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:37,534 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:37,538 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:37,538 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:37,538 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:37,538 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. started... 2023-04-08 20:56:37,542 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ebdeb8625c2b4ac57e00f30df2ca3028 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-08 20:56:37,542 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=A 2023-04-08 20:56:37,542 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:37,542 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=B 2023-04-08 20:56:37,543 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:37,543 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=C 2023-04-08 20:56:37,543 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:37,551 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=450 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/91be56a2ffbf479d852053c8c382e40d 2023-04-08 20:56:37,580 DEBUG [Thread-1674] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x353e11a8 to 127.0.0.1:52440 2023-04-08 20:56:37,580 DEBUG [Thread-1674] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:56:37,581 DEBUG [Thread-1678] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6635f17a to 127.0.0.1:52440 2023-04-08 20:56:37,581 DEBUG [Thread-1678] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:56:37,582 DEBUG [Thread-1676] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x12f583c6 to 127.0.0.1:52440 2023-04-08 20:56:37,582 DEBUG [Thread-1676] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:56:37,583 DEBUG [Thread-1672] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x07742ca6 to 127.0.0.1:52440 2023-04-08 20:56:37,583 DEBUG [Thread-1672] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:56:37,584 DEBUG [Thread-1680] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x38204078 to 127.0.0.1:52440 2023-04-08 20:56:37,584 DEBUG [Thread-1680] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:56:37,946 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/e500bd00d61a4343b3a91652ab193bdd as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/e500bd00d61a4343b3a91652ab193bdd 2023-04-08 20:56:37,946 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/e1e6d3c5c6454de7b5139fb5026cb890 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/e1e6d3c5c6454de7b5139fb5026cb890 2023-04-08 20:56:37,952 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ebdeb8625c2b4ac57e00f30df2ca3028/B of ebdeb8625c2b4ac57e00f30df2ca3028 into e1e6d3c5c6454de7b5139fb5026cb890(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:56:37,952 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:37,952 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028., storeName=ebdeb8625c2b4ac57e00f30df2ca3028/B, priority=11, startTime=1680987397504; duration=0sec 2023-04-08 20:56:37,952 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ebdeb8625c2b4ac57e00f30df2ca3028/A of ebdeb8625c2b4ac57e00f30df2ca3028 into e500bd00d61a4343b3a91652ab193bdd(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:56:37,952 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:56:37,952 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:37,952 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-08 20:56:37,952 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028., storeName=ebdeb8625c2b4ac57e00f30df2ca3028/A, priority=11, startTime=1680987397504; duration=0sec 2023-04-08 20:56:37,952 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:56:37,954 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61612 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-08 20:56:37,954 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): ebdeb8625c2b4ac57e00f30df2ca3028/C is initiating minor compaction (all files) 2023-04-08 20:56:37,954 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ebdeb8625c2b4ac57e00f30df2ca3028/C in TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:37,954 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/68884a2f8eb94220a0700d674276fcb3, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/eeebdf7078e74be5b9dbb25b2bac4fb4, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/ad281e7e07584c5a9d4aef4c61e2f0be, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/5e0c9912a89f4207ac6db281d00f9e40, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/7cd0514dd5f7433e91ed7747d99d91e6] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp, totalSize=60.2 K 2023-04-08 20:56:37,954 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 68884a2f8eb94220a0700d674276fcb3, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1680987390951 2023-04-08 20:56:37,955 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting eeebdf7078e74be5b9dbb25b2bac4fb4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1680987392091 2023-04-08 20:56:37,955 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting ad281e7e07584c5a9d4aef4c61e2f0be, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1680987394295 2023-04-08 20:56:37,955 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 5e0c9912a89f4207ac6db281d00f9e40, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1680987394931 2023-04-08 20:56:37,956 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 7cd0514dd5f7433e91ed7747d99d91e6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=440, earliestPutTs=1680987396141 2023-04-08 20:56:37,961 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): ebdeb8625c2b4ac57e00f30df2ca3028#C#compaction#296 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:56:37,966 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=450 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/8592b2cd4001480080d229392d94a630 2023-04-08 20:56:38,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on ebdeb8625c2b4ac57e00f30df2ca3028 2023-04-08 20:56:38,311 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. as already flushing 2023-04-08 20:56:38,312 DEBUG [Thread-1669] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ffcf146 to 127.0.0.1:52440 2023-04-08 20:56:38,312 DEBUG [Thread-1669] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:56:38,320 DEBUG [Thread-1665] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x24e1e878 to 127.0.0.1:52440 2023-04-08 20:56:38,320 DEBUG [Thread-1665] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:56:38,331 DEBUG [Thread-1661] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2135e03b to 127.0.0.1:52440 2023-04-08 20:56:38,331 DEBUG [Thread-1661] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:56:38,371 DEBUG [Thread-1663] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f2b45f8 to 127.0.0.1:52440 2023-04-08 20:56:38,371 DEBUG [Thread-1663] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:56:38,376 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/cd097e4f94ac43c594dbc4e1a46a3869 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/cd097e4f94ac43c594dbc4e1a46a3869 2023-04-08 20:56:38,377 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=450 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/580955f92321420f8bf808d63603aa18 2023-04-08 20:56:38,383 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ebdeb8625c2b4ac57e00f30df2ca3028/C of ebdeb8625c2b4ac57e00f30df2ca3028 into cd097e4f94ac43c594dbc4e1a46a3869(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:56:38,383 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:38,383 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028., storeName=ebdeb8625c2b4ac57e00f30df2ca3028/C, priority=11, startTime=1680987397504; duration=0sec 2023-04-08 20:56:38,383 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:56:38,384 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/91be56a2ffbf479d852053c8c382e40d as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/91be56a2ffbf479d852053c8c382e40d 2023-04-08 20:56:38,388 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/91be56a2ffbf479d852053c8c382e40d, entries=150, sequenceid=450, filesize=11.9 K 2023-04-08 20:56:38,389 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/8592b2cd4001480080d229392d94a630 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/8592b2cd4001480080d229392d94a630 2023-04-08 20:56:38,393 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/8592b2cd4001480080d229392d94a630, entries=150, sequenceid=450, filesize=11.9 K 2023-04-08 20:56:38,394 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/580955f92321420f8bf808d63603aa18 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/580955f92321420f8bf808d63603aa18 2023-04-08 20:56:38,397 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/580955f92321420f8bf808d63603aa18, entries=150, sequenceid=450, filesize=11.9 K 2023-04-08 20:56:38,398 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=26.84 KB/27480 for ebdeb8625c2b4ac57e00f30df2ca3028 in 856ms, sequenceid=450, compaction requested=false 2023-04-08 20:56:38,398 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:38,398 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:38,398 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:38,398 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:38,398 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:38,398 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:38,398 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:38,399 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:38,399 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:38,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:38,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:38,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:38,400 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:38,400 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:38,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:38,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:38,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:38,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:38,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:38,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:38,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@174a8724[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:38,401 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:38,401 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:38,402 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:38,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:38,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:38,402 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:38,402 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:38,402 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:38,402 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:38,402 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:38,403 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:38,403 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:38,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:38,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:38,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:38,403 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:38,403 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:38,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:38,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:38,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:38,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:38,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:38,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:38,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:38,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:38,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:38,404 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:38,404 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:38,404 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:38,404 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:38,405 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:38,405 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:38,405 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:38,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:38,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:38,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:38,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:38,405 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:38,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:38,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:38,406 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:38,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:38,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:38,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:38,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:38,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:38,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:38,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:38,407 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:38,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:38,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:38,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:38,408 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:38,408 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:38,408 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:38,408 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:38,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:38,408 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:38,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:38,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:38,408 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:38,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:38,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:38,408 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:38,408 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:38,408 DEBUG [Thread-1671] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:38,408 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:38,409 DEBUG [Thread-1671] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:38,409 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:38,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:38,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:38,418 DEBUG [Thread-1667] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e9921a7 to 127.0.0.1:52440 2023-04-08 20:56:38,418 DEBUG [Thread-1667] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:56:38,509 DEBUG [Thread-1671] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:38,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:38,510 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-08 20:56:38,510 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(394): wrote 19 2023-04-08 20:56:38,510 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(394): wrote 21 2023-04-08 20:56:38,510 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(394): wrote 21 2023-04-08 20:56:38,510 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(394): wrote 289 2023-04-08 20:56:38,510 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(394): wrote 22 2023-04-08 20:56:38,511 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-08 20:56:38,511 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(398): read 5295 2023-04-08 20:56:38,511 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(398): read 5334 2023-04-08 20:56:38,511 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(398): read 5301 2023-04-08 20:56:38,511 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(398): read 5334 2023-04-08 20:56:38,511 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(398): read 5329 2023-04-08 20:56:38,511 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-08 20:56:38,511 INFO [Listener at localhost.localdomain/41823] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-08 20:56:38,511 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x73ef029c to 127.0.0.1:52440 2023-04-08 20:56:38,511 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:56:38,512 INFO [Listener at localhost.localdomain/41823] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-08 20:56:38,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-08 20:56:38,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-08 20:56:38,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-08 20:56:38,519 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680987398519"}]},"ts":"1680987398519"} 2023-04-08 20:56:38,520 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-08 20:56:38,526 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-08 20:56:38,527 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ebdeb8625c2b4ac57e00f30df2ca3028, UNASSIGN}] 2023-04-08 20:56:38,528 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ebdeb8625c2b4ac57e00f30df2ca3028, UNASSIGN 2023-04-08 20:56:38,529 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=ebdeb8625c2b4ac57e00f30df2ca3028, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:38,529 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987398529"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680987398529"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680987398529"}]},"ts":"1680987398529"} 2023-04-08 20:56:38,530 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601}] 2023-04-08 20:56:38,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-08 20:56:38,682 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close ebdeb8625c2b4ac57e00f30df2ca3028 2023-04-08 20:56:38,682 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing ebdeb8625c2b4ac57e00f30df2ca3028, disabling compactions & flushes 2023-04-08 20:56:38,682 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:38,682 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:38,682 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. after waiting 0 ms 2023-04-08 20:56:38,683 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:38,683 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing ebdeb8625c2b4ac57e00f30df2ca3028 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-08 20:56:38,683 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=A 2023-04-08 20:56:38,683 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:38,683 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=B 2023-04-08 20:56:38,683 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:38,683 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ebdeb8625c2b4ac57e00f30df2ca3028, store=C 2023-04-08 20:56:38,683 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:38,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-08 20:56:39,091 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/ecdf85f01b1b4d10978ecd33c53b4f26 2023-04-08 20:56:39,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-08 20:56:39,510 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/c037855d822f4aaf8f045c50b3934946 2023-04-08 20:56:39,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-08 20:56:39,929 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/8e677e444ad74318bdf0cf4d443b0396 2023-04-08 20:56:39,934 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/A/ecdf85f01b1b4d10978ecd33c53b4f26 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/ecdf85f01b1b4d10978ecd33c53b4f26 2023-04-08 20:56:39,939 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/ecdf85f01b1b4d10978ecd33c53b4f26, entries=150, sequenceid=461, filesize=11.9 K 2023-04-08 20:56:39,940 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/B/c037855d822f4aaf8f045c50b3934946 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/c037855d822f4aaf8f045c50b3934946 2023-04-08 20:56:39,945 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/c037855d822f4aaf8f045c50b3934946, entries=150, sequenceid=461, filesize=11.9 K 2023-04-08 20:56:39,946 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/.tmp/C/8e677e444ad74318bdf0cf4d443b0396 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/8e677e444ad74318bdf0cf4d443b0396 2023-04-08 20:56:39,951 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/8e677e444ad74318bdf0cf4d443b0396, entries=150, sequenceid=461, filesize=11.9 K 2023-04-08 20:56:39,952 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for ebdeb8625c2b4ac57e00f30df2ca3028 in 1269ms, sequenceid=461, compaction requested=true 2023-04-08 20:56:39,981 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/9711a2f388074f82a11d9d0a6665281c, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/14fc9486bc5a465a8923f86c5e75be88, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/316178808b204f6e92ac0b364666f9b4, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/e11e43423d5840b6a82356f36ba9b352, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/320490a1920e47d4ba982f867f78692e, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/243009136427467c8c96a1655770570f, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/bbf9337be92148f4bcf0dc820d7390c8, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/b442862c6ac84445b7f7063885d859b4, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/de211dcb0b264fa29a320d6f603b64f2, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/dbc957444c8945cd8fe4635a56267437, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/da48d5fd36794e5e9feb1ca4c3fffb44, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/1be3c4f2b9de4e638a806ba98c4af784, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/81e85d1b5e214c4595ad37575b5e145d, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/015077187ac54cf48bab725a8e89b705, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/ea88b23bcb0f401e9e5461d707f55f05, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/e9858080f6e941c5b59f996e82d0c443, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/83e2f2aa1ecd4324b9465a8a099b845a, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/42569f0ad01244b78e57474fbd8e7196, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/788218e7680b42c98dd2c52211606911, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/cda2ad3a1fb74bb7b9577e75585ddcb9, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/3cf5514e5c074f3ebbba51ddf7513a54, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/d42738549e414fd0917ea5bdd5c58e23, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/eaeb7153c74b42b887c30c0f90b4d185, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/e3ad4eeebce84388813f48dfdb0f1af2, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/31e755da37364f0da7729b756347925b, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/ce1b4d93432a487f9d60a87063ee8629, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/3a059bf715c24a5589dadc6fcea9aaaf] to archive 2023-04-08 20:56:39,982 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 20:56:39,983 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/9711a2f388074f82a11d9d0a6665281c to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/9711a2f388074f82a11d9d0a6665281c 2023-04-08 20:56:39,985 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/14fc9486bc5a465a8923f86c5e75be88 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/14fc9486bc5a465a8923f86c5e75be88 2023-04-08 20:56:39,986 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/316178808b204f6e92ac0b364666f9b4 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/316178808b204f6e92ac0b364666f9b4 2023-04-08 20:56:39,987 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/e11e43423d5840b6a82356f36ba9b352 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/e11e43423d5840b6a82356f36ba9b352 2023-04-08 20:56:39,988 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/320490a1920e47d4ba982f867f78692e to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/320490a1920e47d4ba982f867f78692e 2023-04-08 20:56:39,989 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/243009136427467c8c96a1655770570f to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/243009136427467c8c96a1655770570f 2023-04-08 20:56:39,990 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/bbf9337be92148f4bcf0dc820d7390c8 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/bbf9337be92148f4bcf0dc820d7390c8 2023-04-08 20:56:39,990 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/b442862c6ac84445b7f7063885d859b4 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/b442862c6ac84445b7f7063885d859b4 2023-04-08 20:56:39,991 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/de211dcb0b264fa29a320d6f603b64f2 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/de211dcb0b264fa29a320d6f603b64f2 2023-04-08 20:56:39,992 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/dbc957444c8945cd8fe4635a56267437 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/dbc957444c8945cd8fe4635a56267437 2023-04-08 20:56:39,993 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/da48d5fd36794e5e9feb1ca4c3fffb44 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/da48d5fd36794e5e9feb1ca4c3fffb44 2023-04-08 20:56:39,994 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/1be3c4f2b9de4e638a806ba98c4af784 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/1be3c4f2b9de4e638a806ba98c4af784 2023-04-08 20:56:39,995 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/81e85d1b5e214c4595ad37575b5e145d to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/81e85d1b5e214c4595ad37575b5e145d 2023-04-08 20:56:39,995 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/015077187ac54cf48bab725a8e89b705 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/015077187ac54cf48bab725a8e89b705 2023-04-08 20:56:39,996 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/ea88b23bcb0f401e9e5461d707f55f05 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/ea88b23bcb0f401e9e5461d707f55f05 2023-04-08 20:56:39,997 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/e9858080f6e941c5b59f996e82d0c443 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/e9858080f6e941c5b59f996e82d0c443 2023-04-08 20:56:39,998 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/83e2f2aa1ecd4324b9465a8a099b845a to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/83e2f2aa1ecd4324b9465a8a099b845a 2023-04-08 20:56:39,998 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/42569f0ad01244b78e57474fbd8e7196 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/42569f0ad01244b78e57474fbd8e7196 2023-04-08 20:56:39,999 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/788218e7680b42c98dd2c52211606911 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/788218e7680b42c98dd2c52211606911 2023-04-08 20:56:40,000 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/cda2ad3a1fb74bb7b9577e75585ddcb9 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/cda2ad3a1fb74bb7b9577e75585ddcb9 2023-04-08 20:56:40,000 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/3cf5514e5c074f3ebbba51ddf7513a54 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/3cf5514e5c074f3ebbba51ddf7513a54 2023-04-08 20:56:40,001 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/d42738549e414fd0917ea5bdd5c58e23 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/d42738549e414fd0917ea5bdd5c58e23 2023-04-08 20:56:40,002 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/eaeb7153c74b42b887c30c0f90b4d185 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/eaeb7153c74b42b887c30c0f90b4d185 2023-04-08 20:56:40,003 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/e3ad4eeebce84388813f48dfdb0f1af2 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/e3ad4eeebce84388813f48dfdb0f1af2 2023-04-08 20:56:40,003 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/31e755da37364f0da7729b756347925b to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/31e755da37364f0da7729b756347925b 2023-04-08 20:56:40,004 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/ce1b4d93432a487f9d60a87063ee8629 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/ce1b4d93432a487f9d60a87063ee8629 2023-04-08 20:56:40,005 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/3a059bf715c24a5589dadc6fcea9aaaf to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/3a059bf715c24a5589dadc6fcea9aaaf 2023-04-08 20:56:40,010 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/dd22993dfba649ac9987823e17a91ec5, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/ecdbe5c948e640ddaf0ad076a0aa31a3, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/3d38a91e832e4038bc76664e38f02d8e, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/aa83af904119442f81ef55453a956583, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/9e7058a141394626bccfd63000b9c688, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/58b971b7f8154748b32aa8568640d49e, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/22bba28af2d74afc8929b38072562408, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/5e30a0ab3e4f4a05986f819531c9e577, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/3fe398aa3624424181e0724599ec72fb, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/ee3167aca4e549c6902d3e39c7458d40, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/8fd130d9f8ee4bac98c987b38e0b4577, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/45b7a8ff355440cd9ad18b9a41445ced, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/99fad3c46b484d2b88474a22f8b40109, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/d735dee4216842d9aacf3a512ca201fc, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/90f0baed13a341248a7a7db09aa53731, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/ddf855bf797247f69ad8685dc61cefd6, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/00f2634921d8410fb9224df22bf2bfbb, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/2a1c135d15024dd2bb39145279b21470, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/e3990c72148940b9aa72053582507e8b, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/1e88adf60591498697b0a402312bec89, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/b10b0b4ec47f4669ae501870e28dc848, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/bd74065e3eb94d3b92ccc230b0a5dd9d, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/b4b76e865899497d8c93643246097850, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/1612c51892c0456ea5d07002479f1bea, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/6ac6acc784e742af8d2bf43e1a587e8b, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/aebc8b92324f48ccac3dbeafe0a517d1, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/97340d0fe2e54eb684528437f8986a45] to archive 2023-04-08 20:56:40,011 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 20:56:40,012 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/dd22993dfba649ac9987823e17a91ec5 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/dd22993dfba649ac9987823e17a91ec5 2023-04-08 20:56:40,013 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/ecdbe5c948e640ddaf0ad076a0aa31a3 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/ecdbe5c948e640ddaf0ad076a0aa31a3 2023-04-08 20:56:40,014 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/3d38a91e832e4038bc76664e38f02d8e to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/3d38a91e832e4038bc76664e38f02d8e 2023-04-08 20:56:40,015 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/aa83af904119442f81ef55453a956583 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/aa83af904119442f81ef55453a956583 2023-04-08 20:56:40,016 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/9e7058a141394626bccfd63000b9c688 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/9e7058a141394626bccfd63000b9c688 2023-04-08 20:56:40,017 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/58b971b7f8154748b32aa8568640d49e to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/58b971b7f8154748b32aa8568640d49e 2023-04-08 20:56:40,018 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/22bba28af2d74afc8929b38072562408 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/22bba28af2d74afc8929b38072562408 2023-04-08 20:56:40,019 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/5e30a0ab3e4f4a05986f819531c9e577 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/5e30a0ab3e4f4a05986f819531c9e577 2023-04-08 20:56:40,020 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/3fe398aa3624424181e0724599ec72fb to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/3fe398aa3624424181e0724599ec72fb 2023-04-08 20:56:40,021 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/ee3167aca4e549c6902d3e39c7458d40 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/ee3167aca4e549c6902d3e39c7458d40 2023-04-08 20:56:40,023 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/8fd130d9f8ee4bac98c987b38e0b4577 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/8fd130d9f8ee4bac98c987b38e0b4577 2023-04-08 20:56:40,024 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/45b7a8ff355440cd9ad18b9a41445ced to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/45b7a8ff355440cd9ad18b9a41445ced 2023-04-08 20:56:40,025 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/99fad3c46b484d2b88474a22f8b40109 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/99fad3c46b484d2b88474a22f8b40109 2023-04-08 20:56:40,026 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/d735dee4216842d9aacf3a512ca201fc to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/d735dee4216842d9aacf3a512ca201fc 2023-04-08 20:56:40,027 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/90f0baed13a341248a7a7db09aa53731 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/90f0baed13a341248a7a7db09aa53731 2023-04-08 20:56:40,029 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/ddf855bf797247f69ad8685dc61cefd6 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/ddf855bf797247f69ad8685dc61cefd6 2023-04-08 20:56:40,030 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/00f2634921d8410fb9224df22bf2bfbb to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/00f2634921d8410fb9224df22bf2bfbb 2023-04-08 20:56:40,032 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/2a1c135d15024dd2bb39145279b21470 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/2a1c135d15024dd2bb39145279b21470 2023-04-08 20:56:40,033 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/e3990c72148940b9aa72053582507e8b to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/e3990c72148940b9aa72053582507e8b 2023-04-08 20:56:40,035 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/1e88adf60591498697b0a402312bec89 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/1e88adf60591498697b0a402312bec89 2023-04-08 20:56:40,035 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/b10b0b4ec47f4669ae501870e28dc848 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/b10b0b4ec47f4669ae501870e28dc848 2023-04-08 20:56:40,036 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/bd74065e3eb94d3b92ccc230b0a5dd9d to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/bd74065e3eb94d3b92ccc230b0a5dd9d 2023-04-08 20:56:40,037 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/b4b76e865899497d8c93643246097850 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/b4b76e865899497d8c93643246097850 2023-04-08 20:56:40,038 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/1612c51892c0456ea5d07002479f1bea to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/1612c51892c0456ea5d07002479f1bea 2023-04-08 20:56:40,039 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/6ac6acc784e742af8d2bf43e1a587e8b to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/6ac6acc784e742af8d2bf43e1a587e8b 2023-04-08 20:56:40,040 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/aebc8b92324f48ccac3dbeafe0a517d1 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/aebc8b92324f48ccac3dbeafe0a517d1 2023-04-08 20:56:40,041 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/97340d0fe2e54eb684528437f8986a45 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/97340d0fe2e54eb684528437f8986a45 2023-04-08 20:56:40,044 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/db9d492ec4994ce9b2ec51cd5a8e599c, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/fb70b25972cb44f3be382464c4bb7685, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/91202d7d40ac45afbfe6344114aeec53, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/f692e50ad2184121b1175edb4db4e42d, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/667d8062e47d4252b66948dfbd2fc0da, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/4969367a170d4f47bdcf9468b9cb3c67, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/314678b7962340c3ba02e10fd0f7ed5b, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/f32dfaac5bfe4f0a8e2d669c264e6f84, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/7fa87866a6654accb0cf57a730b16e41, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/2839ef00ae7e4be4a1dfb6483b9a9fa9, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/e6ed48e71d4c459a820bdc67a1fe2b8e, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/e25c9557abab4a3cbffcc240a86bacc1, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/db04c203fb8545d3a10313f0972a119d, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/334261fcc3c54c62bc6de484c8b69b1d, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/c6aadbd82cb54dc4b86ab68af81766e5, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/cdc4f6ac56fb4349a011c0d431fd5368, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/9327d46b760749cf82d7fed39793361b, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/ecb18c00d31c4b1a839eaf791be7ebfe, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/5a64000fc6874832ba507f46d4d2f715, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/68884a2f8eb94220a0700d674276fcb3, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/9d90d95b0a2f46198af639d009596b63, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/eeebdf7078e74be5b9dbb25b2bac4fb4, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/ad281e7e07584c5a9d4aef4c61e2f0be, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/5e0c9912a89f4207ac6db281d00f9e40, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/7cd0514dd5f7433e91ed7747d99d91e6] to archive 2023-04-08 20:56:40,044 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 20:56:40,045 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/db9d492ec4994ce9b2ec51cd5a8e599c to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/db9d492ec4994ce9b2ec51cd5a8e599c 2023-04-08 20:56:40,046 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/fb70b25972cb44f3be382464c4bb7685 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/fb70b25972cb44f3be382464c4bb7685 2023-04-08 20:56:40,047 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/91202d7d40ac45afbfe6344114aeec53 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/91202d7d40ac45afbfe6344114aeec53 2023-04-08 20:56:40,048 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/f692e50ad2184121b1175edb4db4e42d to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/f692e50ad2184121b1175edb4db4e42d 2023-04-08 20:56:40,049 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/667d8062e47d4252b66948dfbd2fc0da to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/667d8062e47d4252b66948dfbd2fc0da 2023-04-08 20:56:40,050 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/4969367a170d4f47bdcf9468b9cb3c67 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/4969367a170d4f47bdcf9468b9cb3c67 2023-04-08 20:56:40,051 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/314678b7962340c3ba02e10fd0f7ed5b to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/314678b7962340c3ba02e10fd0f7ed5b 2023-04-08 20:56:40,051 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/f32dfaac5bfe4f0a8e2d669c264e6f84 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/f32dfaac5bfe4f0a8e2d669c264e6f84 2023-04-08 20:56:40,052 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/7fa87866a6654accb0cf57a730b16e41 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/7fa87866a6654accb0cf57a730b16e41 2023-04-08 20:56:40,053 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/2839ef00ae7e4be4a1dfb6483b9a9fa9 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/2839ef00ae7e4be4a1dfb6483b9a9fa9 2023-04-08 20:56:40,054 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/e6ed48e71d4c459a820bdc67a1fe2b8e to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/e6ed48e71d4c459a820bdc67a1fe2b8e 2023-04-08 20:56:40,054 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/e25c9557abab4a3cbffcc240a86bacc1 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/e25c9557abab4a3cbffcc240a86bacc1 2023-04-08 20:56:40,055 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/db04c203fb8545d3a10313f0972a119d to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/db04c203fb8545d3a10313f0972a119d 2023-04-08 20:56:40,056 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/334261fcc3c54c62bc6de484c8b69b1d to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/334261fcc3c54c62bc6de484c8b69b1d 2023-04-08 20:56:40,057 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/c6aadbd82cb54dc4b86ab68af81766e5 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/c6aadbd82cb54dc4b86ab68af81766e5 2023-04-08 20:56:40,058 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/cdc4f6ac56fb4349a011c0d431fd5368 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/cdc4f6ac56fb4349a011c0d431fd5368 2023-04-08 20:56:40,059 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/9327d46b760749cf82d7fed39793361b to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/9327d46b760749cf82d7fed39793361b 2023-04-08 20:56:40,060 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/ecb18c00d31c4b1a839eaf791be7ebfe to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/ecb18c00d31c4b1a839eaf791be7ebfe 2023-04-08 20:56:40,061 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/5a64000fc6874832ba507f46d4d2f715 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/5a64000fc6874832ba507f46d4d2f715 2023-04-08 20:56:40,062 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/68884a2f8eb94220a0700d674276fcb3 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/68884a2f8eb94220a0700d674276fcb3 2023-04-08 20:56:40,063 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/9d90d95b0a2f46198af639d009596b63 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/9d90d95b0a2f46198af639d009596b63 2023-04-08 20:56:40,064 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/eeebdf7078e74be5b9dbb25b2bac4fb4 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/eeebdf7078e74be5b9dbb25b2bac4fb4 2023-04-08 20:56:40,065 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/ad281e7e07584c5a9d4aef4c61e2f0be to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/ad281e7e07584c5a9d4aef4c61e2f0be 2023-04-08 20:56:40,065 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/5e0c9912a89f4207ac6db281d00f9e40 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/5e0c9912a89f4207ac6db281d00f9e40 2023-04-08 20:56:40,066 DEBUG [StoreCloser-TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/7cd0514dd5f7433e91ed7747d99d91e6 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/7cd0514dd5f7433e91ed7747d99d91e6 2023-04-08 20:56:40,069 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/recovered.edits/464.seqid, newMaxSeqId=464, maxSeqId=1 2023-04-08 20:56:40,070 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028. 2023-04-08 20:56:40,070 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for ebdeb8625c2b4ac57e00f30df2ca3028: 2023-04-08 20:56:40,072 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed ebdeb8625c2b4ac57e00f30df2ca3028 2023-04-08 20:56:40,072 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=ebdeb8625c2b4ac57e00f30df2ca3028, regionState=CLOSED 2023-04-08 20:56:40,072 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987400072"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680987400072"}]},"ts":"1680987400072"} 2023-04-08 20:56:40,074 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-08 20:56:40,074 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure ebdeb8625c2b4ac57e00f30df2ca3028, server=jenkins-hbase20.apache.org,41091,1680987313601 in 1.5430 sec 2023-04-08 20:56:40,076 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-08 20:56:40,076 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=ebdeb8625c2b4ac57e00f30df2ca3028, UNASSIGN in 1.5470 sec 2023-04-08 20:56:40,076 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680987400076"}]},"ts":"1680987400076"} 2023-04-08 20:56:40,077 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-08 20:56:40,078 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-08 20:56:40,080 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5660 sec 2023-04-08 20:56:40,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-08 20:56:40,626 INFO [Listener at localhost.localdomain/41823] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-08 20:56:40,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-08 20:56:40,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 20:56:40,629 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 20:56:40,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-08 20:56:40,630 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 20:56:40,635 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028 2023-04-08 20:56:40,637 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A, FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B, FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C, FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/recovered.edits] 2023-04-08 20:56:40,640 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/91be56a2ffbf479d852053c8c382e40d to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/91be56a2ffbf479d852053c8c382e40d 2023-04-08 20:56:40,641 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/e500bd00d61a4343b3a91652ab193bdd to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/e500bd00d61a4343b3a91652ab193bdd 2023-04-08 20:56:40,642 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/ecdf85f01b1b4d10978ecd33c53b4f26 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/A/ecdf85f01b1b4d10978ecd33c53b4f26 2023-04-08 20:56:40,644 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/8592b2cd4001480080d229392d94a630 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/8592b2cd4001480080d229392d94a630 2023-04-08 20:56:40,645 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/c037855d822f4aaf8f045c50b3934946 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/c037855d822f4aaf8f045c50b3934946 2023-04-08 20:56:40,646 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/e1e6d3c5c6454de7b5139fb5026cb890 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/B/e1e6d3c5c6454de7b5139fb5026cb890 2023-04-08 20:56:40,647 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/580955f92321420f8bf808d63603aa18 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/580955f92321420f8bf808d63603aa18 2023-04-08 20:56:40,648 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/8e677e444ad74318bdf0cf4d443b0396 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/8e677e444ad74318bdf0cf4d443b0396 2023-04-08 20:56:40,649 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/cd097e4f94ac43c594dbc4e1a46a3869 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/C/cd097e4f94ac43c594dbc4e1a46a3869 2023-04-08 20:56:40,651 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/recovered.edits/464.seqid to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028/recovered.edits/464.seqid 2023-04-08 20:56:40,651 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/ebdeb8625c2b4ac57e00f30df2ca3028 2023-04-08 20:56:40,651 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-08 20:56:40,653 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 20:56:40,654 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-08 20:56:40,655 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-08 20:56:40,656 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 20:56:40,656 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-08 20:56:40,656 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680987400656"}]},"ts":"9223372036854775807"} 2023-04-08 20:56:40,658 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-08 20:56:40,658 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => ebdeb8625c2b4ac57e00f30df2ca3028, NAME => 'TestAcidGuarantees,,1680987375363.ebdeb8625c2b4ac57e00f30df2ca3028.', STARTKEY => '', ENDKEY => ''}] 2023-04-08 20:56:40,658 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-08 20:56:40,658 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680987400658"}]},"ts":"9223372036854775807"} 2023-04-08 20:56:40,659 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-08 20:56:40,660 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 20:56:40,661 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 33 msec 2023-04-08 20:56:40,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-08 20:56:40,731 INFO [Listener at localhost.localdomain/41823] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-08 20:56:40,744 INFO [Listener at localhost.localdomain/41823] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=229 (was 228) - Thread LEAK? -, OpenFileDescriptor=531 (was 550), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=367 (was 320) - SystemLoadAverage LEAK? -, ProcessCount=166 (was 166), AvailableMemoryMB=3460 (was 3916) 2023-04-08 20:56:40,755 INFO [Listener at localhost.localdomain/41823] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=229, OpenFileDescriptor=531, MaxFileDescriptor=60000, SystemLoadAverage=367, ProcessCount=166, AvailableMemoryMB=3459 2023-04-08 20:56:40,757 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-08 20:56:40,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-08 20:56:40,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-08 20:56:40,760 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-08 20:56:40,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-08 20:56:40,761 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-08 20:56:40,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-08 20:56:40,762 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:56:40,762 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc empty. 2023-04-08 20:56:40,763 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:56:40,763 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-08 20:56:40,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-08 20:56:41,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-08 20:56:41,171 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-08 20:56:41,174 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => eb4c5ed1339ded9a0ba1af8980e65ffc, NAME => 'TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp 2023-04-08 20:56:41,189 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 20:56:41,190 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing eb4c5ed1339ded9a0ba1af8980e65ffc, disabling compactions & flushes 2023-04-08 20:56:41,190 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:41,190 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:41,190 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. after waiting 0 ms 2023-04-08 20:56:41,190 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:41,190 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:41,190 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for eb4c5ed1339ded9a0ba1af8980e65ffc: 2023-04-08 20:56:41,192 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-08 20:56:41,193 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987401193"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680987401193"}]},"ts":"1680987401193"} 2023-04-08 20:56:41,194 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-08 20:56:41,195 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-08 20:56:41,195 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680987401195"}]},"ts":"1680987401195"} 2023-04-08 20:56:41,195 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-08 20:56:41,197 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=eb4c5ed1339ded9a0ba1af8980e65ffc, ASSIGN}] 2023-04-08 20:56:41,199 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=eb4c5ed1339ded9a0ba1af8980e65ffc, ASSIGN 2023-04-08 20:56:41,199 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=eb4c5ed1339ded9a0ba1af8980e65ffc, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,41091,1680987313601; forceNewPlan=false, retain=false 2023-04-08 20:56:41,350 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=eb4c5ed1339ded9a0ba1af8980e65ffc, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:41,351 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987401350"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680987401350"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680987401350"}]},"ts":"1680987401350"} 2023-04-08 20:56:41,352 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601}] 2023-04-08 20:56:41,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-08 20:56:41,509 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:41,509 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => eb4c5ed1339ded9a0ba1af8980e65ffc, NAME => 'TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.', STARTKEY => '', ENDKEY => ''} 2023-04-08 20:56:41,510 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:56:41,510 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 20:56:41,510 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:56:41,510 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:56:41,512 INFO [StoreOpener-eb4c5ed1339ded9a0ba1af8980e65ffc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:56:41,515 DEBUG [StoreOpener-eb4c5ed1339ded9a0ba1af8980e65ffc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A 2023-04-08 20:56:41,515 DEBUG [StoreOpener-eb4c5ed1339ded9a0ba1af8980e65ffc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A 2023-04-08 20:56:41,515 INFO [StoreOpener-eb4c5ed1339ded9a0ba1af8980e65ffc-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 20:56:41,516 INFO [StoreOpener-eb4c5ed1339ded9a0ba1af8980e65ffc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region eb4c5ed1339ded9a0ba1af8980e65ffc columnFamilyName A 2023-04-08 20:56:41,517 INFO [StoreOpener-eb4c5ed1339ded9a0ba1af8980e65ffc-1] regionserver.HStore(310): Store=eb4c5ed1339ded9a0ba1af8980e65ffc/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 20:56:41,517 INFO [StoreOpener-eb4c5ed1339ded9a0ba1af8980e65ffc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:56:41,519 DEBUG [StoreOpener-eb4c5ed1339ded9a0ba1af8980e65ffc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B 2023-04-08 20:56:41,520 DEBUG [StoreOpener-eb4c5ed1339ded9a0ba1af8980e65ffc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B 2023-04-08 20:56:41,520 INFO [StoreOpener-eb4c5ed1339ded9a0ba1af8980e65ffc-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 20:56:41,520 INFO [StoreOpener-eb4c5ed1339ded9a0ba1af8980e65ffc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region eb4c5ed1339ded9a0ba1af8980e65ffc columnFamilyName B 2023-04-08 20:56:41,521 INFO [StoreOpener-eb4c5ed1339ded9a0ba1af8980e65ffc-1] regionserver.HStore(310): Store=eb4c5ed1339ded9a0ba1af8980e65ffc/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 20:56:41,521 INFO [StoreOpener-eb4c5ed1339ded9a0ba1af8980e65ffc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:56:41,523 DEBUG [StoreOpener-eb4c5ed1339ded9a0ba1af8980e65ffc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C 2023-04-08 20:56:41,523 DEBUG [StoreOpener-eb4c5ed1339ded9a0ba1af8980e65ffc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C 2023-04-08 20:56:41,524 INFO [StoreOpener-eb4c5ed1339ded9a0ba1af8980e65ffc-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 20:56:41,524 INFO [StoreOpener-eb4c5ed1339ded9a0ba1af8980e65ffc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region eb4c5ed1339ded9a0ba1af8980e65ffc columnFamilyName C 2023-04-08 20:56:41,525 INFO [StoreOpener-eb4c5ed1339ded9a0ba1af8980e65ffc-1] regionserver.HStore(310): Store=eb4c5ed1339ded9a0ba1af8980e65ffc/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 20:56:41,525 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:41,525 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:56:41,526 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:56:41,528 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-08 20:56:41,529 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:56:41,531 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-08 20:56:41,531 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened eb4c5ed1339ded9a0ba1af8980e65ffc; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71989387, jitterRate=0.07272545993328094}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-08 20:56:41,531 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for eb4c5ed1339ded9a0ba1af8980e65ffc: 2023-04-08 20:56:41,532 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc., pid=37, masterSystemTime=1680987401503 2023-04-08 20:56:41,533 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:41,533 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:41,534 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=eb4c5ed1339ded9a0ba1af8980e65ffc, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:41,534 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987401534"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680987401534"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680987401534"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680987401534"}]},"ts":"1680987401534"} 2023-04-08 20:56:41,536 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-08 20:56:41,537 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 in 183 msec 2023-04-08 20:56:41,538 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-08 20:56:41,539 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=eb4c5ed1339ded9a0ba1af8980e65ffc, ASSIGN in 340 msec 2023-04-08 20:56:41,539 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-08 20:56:41,539 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680987401539"}]},"ts":"1680987401539"} 2023-04-08 20:56:41,541 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-08 20:56:41,543 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-08 20:56:41,544 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 786 msec 2023-04-08 20:56:41,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-08 20:56:41,868 INFO [Listener at localhost.localdomain/41823] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-08 20:56:41,869 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x6f29a773 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:56:41,878 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1b6e51f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:56:41,880 DEBUG [Listener at localhost.localdomain/41823] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:56:41,884 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54722, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:56:41,885 DEBUG [Listener at localhost.localdomain/41823] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-08 20:56:41,887 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:33948, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-08 20:56:41,889 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-08 20:56:41,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-08 20:56:41,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-08 20:56:42,302 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-08 20:56:42,304 DEBUG [PEWorker-5] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-08 20:56:42,305 INFO [PEWorker-5] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-08 20:56:42,309 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-08 20:56:42,312 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=eb4c5ed1339ded9a0ba1af8980e65ffc, REOPEN/MOVE}] 2023-04-08 20:56:42,313 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=eb4c5ed1339ded9a0ba1af8980e65ffc, REOPEN/MOVE 2023-04-08 20:56:42,314 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=eb4c5ed1339ded9a0ba1af8980e65ffc, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:42,314 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987402314"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680987402314"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680987402314"}]},"ts":"1680987402314"} 2023-04-08 20:56:42,315 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601}] 2023-04-08 20:56:42,468 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:56:42,469 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing eb4c5ed1339ded9a0ba1af8980e65ffc, disabling compactions & flushes 2023-04-08 20:56:42,469 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:42,469 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:42,469 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. after waiting 0 ms 2023-04-08 20:56:42,469 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:42,477 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-08 20:56:42,478 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:42,478 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for eb4c5ed1339ded9a0ba1af8980e65ffc: 2023-04-08 20:56:42,478 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: eb4c5ed1339ded9a0ba1af8980e65ffc to self. 2023-04-08 20:56:42,481 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:56:42,481 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=eb4c5ed1339ded9a0ba1af8980e65ffc, regionState=CLOSED 2023-04-08 20:56:42,481 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987402481"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680987402481"}]},"ts":"1680987402481"} 2023-04-08 20:56:42,485 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-08 20:56:42,485 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 in 168 msec 2023-04-08 20:56:42,486 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=eb4c5ed1339ded9a0ba1af8980e65ffc, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,41091,1680987313601; forceNewPlan=false, retain=true 2023-04-08 20:56:42,637 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=eb4c5ed1339ded9a0ba1af8980e65ffc, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:42,637 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987402637"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680987402637"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680987402637"}]},"ts":"1680987402637"} 2023-04-08 20:56:42,639 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601}] 2023-04-08 20:56:42,799 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:42,800 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => eb4c5ed1339ded9a0ba1af8980e65ffc, NAME => 'TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.', STARTKEY => '', ENDKEY => ''} 2023-04-08 20:56:42,801 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:56:42,801 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 20:56:42,801 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:56:42,801 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:56:42,803 INFO [StoreOpener-eb4c5ed1339ded9a0ba1af8980e65ffc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:56:42,804 DEBUG [StoreOpener-eb4c5ed1339ded9a0ba1af8980e65ffc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A 2023-04-08 20:56:42,804 DEBUG [StoreOpener-eb4c5ed1339ded9a0ba1af8980e65ffc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A 2023-04-08 20:56:42,805 INFO [StoreOpener-eb4c5ed1339ded9a0ba1af8980e65ffc-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 20:56:42,805 INFO [StoreOpener-eb4c5ed1339ded9a0ba1af8980e65ffc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region eb4c5ed1339ded9a0ba1af8980e65ffc columnFamilyName A 2023-04-08 20:56:42,807 INFO [StoreOpener-eb4c5ed1339ded9a0ba1af8980e65ffc-1] regionserver.HStore(310): Store=eb4c5ed1339ded9a0ba1af8980e65ffc/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 20:56:42,807 INFO [StoreOpener-eb4c5ed1339ded9a0ba1af8980e65ffc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:56:42,808 DEBUG [StoreOpener-eb4c5ed1339ded9a0ba1af8980e65ffc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B 2023-04-08 20:56:42,808 DEBUG [StoreOpener-eb4c5ed1339ded9a0ba1af8980e65ffc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B 2023-04-08 20:56:42,808 INFO [StoreOpener-eb4c5ed1339ded9a0ba1af8980e65ffc-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 20:56:42,808 INFO [StoreOpener-eb4c5ed1339ded9a0ba1af8980e65ffc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region eb4c5ed1339ded9a0ba1af8980e65ffc columnFamilyName B 2023-04-08 20:56:42,809 INFO [StoreOpener-eb4c5ed1339ded9a0ba1af8980e65ffc-1] regionserver.HStore(310): Store=eb4c5ed1339ded9a0ba1af8980e65ffc/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 20:56:42,809 INFO [StoreOpener-eb4c5ed1339ded9a0ba1af8980e65ffc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:56:42,810 DEBUG [StoreOpener-eb4c5ed1339ded9a0ba1af8980e65ffc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C 2023-04-08 20:56:42,810 DEBUG [StoreOpener-eb4c5ed1339ded9a0ba1af8980e65ffc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C 2023-04-08 20:56:42,810 INFO [StoreOpener-eb4c5ed1339ded9a0ba1af8980e65ffc-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 20:56:42,810 INFO [StoreOpener-eb4c5ed1339ded9a0ba1af8980e65ffc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region eb4c5ed1339ded9a0ba1af8980e65ffc columnFamilyName C 2023-04-08 20:56:42,810 INFO [StoreOpener-eb4c5ed1339ded9a0ba1af8980e65ffc-1] regionserver.HStore(310): Store=eb4c5ed1339ded9a0ba1af8980e65ffc/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 20:56:42,810 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:42,811 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:56:42,812 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:56:42,814 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-08 20:56:42,815 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:56:42,816 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened eb4c5ed1339ded9a0ba1af8980e65ffc; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67847718, jitterRate=0.011009782552719116}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-08 20:56:42,816 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for eb4c5ed1339ded9a0ba1af8980e65ffc: 2023-04-08 20:56:42,816 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc., pid=42, masterSystemTime=1680987402792 2023-04-08 20:56:42,817 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:42,817 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:42,818 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=eb4c5ed1339ded9a0ba1af8980e65ffc, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:42,818 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987402818"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680987402818"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680987402818"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680987402818"}]},"ts":"1680987402818"} 2023-04-08 20:56:42,820 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-08 20:56:42,820 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 in 180 msec 2023-04-08 20:56:42,821 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-08 20:56:42,821 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=eb4c5ed1339ded9a0ba1af8980e65ffc, REOPEN/MOVE in 508 msec 2023-04-08 20:56:42,822 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-08 20:56:42,822 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 512 msec 2023-04-08 20:56:42,823 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 933 msec 2023-04-08 20:56:42,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-08 20:56:42,825 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x48681e8f to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:56:42,828 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6ad83ace, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:56:42,828 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x342c272f to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:56:42,832 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3cb8db43, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:56:42,832 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x4e90d228 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:56:42,836 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1bd0918a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:56:42,836 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x1aec6811 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:56:42,840 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5225f8f6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:56:42,841 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x6d951686 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:56:42,852 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@144047c5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:56:42,852 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x525c59d1 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:56:42,864 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@44b44230, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:56:42,864 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x078ba718 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:56:42,868 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@37a9f4e9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:56:42,868 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x1747f9fc to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:56:42,880 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4d2e2f6b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:56:42,881 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x3098b23b to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:56:42,885 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@183835cd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:56:42,885 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x55dc208b to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:56:42,890 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@20a04961, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:56:42,898 DEBUG [hconnection-0x175f84ce-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:56:42,899 DEBUG [hconnection-0x1f25088a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:56:42,900 DEBUG [hconnection-0x24f0d38c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:56:42,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:42,900 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54738, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:56:42,900 DEBUG [hconnection-0x5de9d5e2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:56:42,902 DEBUG [hconnection-0x1b5c6416-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:56:42,902 DEBUG [hconnection-0x3e72c63a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:56:42,902 DEBUG [hconnection-0x499120f6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:56:42,902 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54766, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:56:42,902 DEBUG [hconnection-0x217eb140-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:56:42,908 DEBUG [hconnection-0x495b9b5d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:56:42,908 DEBUG [hconnection-0x631bdeb3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:56:42,909 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54752, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:56:42,909 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54784, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:56:42,909 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54750, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:56:42,911 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54780, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:56:42,911 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54772, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:56:42,913 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54810, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:56:42,918 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54798, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:56:42,919 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54794, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:56:42,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:42,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:42,930 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:42,932 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:42,932 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:42,932 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:42,933 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:42,933 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:42,934 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:42,935 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:42,935 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:42,936 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:42,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:42,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:42,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:42,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:42,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:42,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:42,939 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:42,940 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing eb4c5ed1339ded9a0ba1af8980e65ffc 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-08 20:56:42,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:56:42,940 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:42,940 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:42,940 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:42,940 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:42,940 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:42,941 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=A 2023-04-08 20:56:42,941 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:42,941 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:42,941 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:42,941 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:42,941 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:56:42,941 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=B 2023-04-08 20:56:42,941 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:56:42,941 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:42,941 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:42,941 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=C 2023-04-08 20:56:42,941 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:42,941 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:42,941 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:42,941 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:42,941 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:42,941 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:42,942 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:42,942 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:42,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:42,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:42,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:42,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:42,943 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:42,943 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:42,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:42,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:42,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:42,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:42,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:42,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76926533[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:42,944 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:42,945 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:42,946 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:42,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:42,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:42,946 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:42,946 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:42,946 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:42,946 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:42,946 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:42,947 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:42,947 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:42,947 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:42,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:42,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:42,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:42,947 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:42,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:42,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:42,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:42,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:42,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:42,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:42,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:42,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:42,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:42,951 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:42,951 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:42,951 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:42,951 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:42,951 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:42,952 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:42,952 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:42,952 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:42,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:42,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:42,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:42,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:42,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:42,953 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:42,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:42,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:42,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:42,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:42,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:42,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:42,954 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:42,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:42,954 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:42,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:42,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:42,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987462953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:42,954 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:42,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:42,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987462954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:42,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987462954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:42,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987462953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:42,955 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:42,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987462954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:42,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:42,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:42,955 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:42,955 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:42,955 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:42,955 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:42,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:42,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:42,956 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:42,956 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:42,955 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:42,956 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:42,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:42,956 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:42,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:42,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:42,956 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:42,956 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:42,956 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:42,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:42,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:42,980 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408e5703a2fa8614f1f991d90487f5ad0f9 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408e5703a2fa8614f1f991d90487f5ad0f9 2023-04-08 20:56:43,056 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:43,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:43,059 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:43,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987463055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,059 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:43,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987463055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,060 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:43,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987463055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,060 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:43,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987463055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,060 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:43,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987463055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:43,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:43,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:43,066 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:43,066 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:43,066 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:43,066 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:43,066 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,066 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,067 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:43,067 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:43,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:43,067 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,067 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:43,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:43,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,068 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:43,069 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:43,069 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:43,069 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:43,069 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:43,069 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:43,069 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:43,069 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:43,069 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:43,069 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:56:43,069 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:56:43,069 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:43,069 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:43,069 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:43,069 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,069 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:43,069 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:43,070 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,070 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:43,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:43,070 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,070 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:43,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:43,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:43,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:43,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:43,071 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:43,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3606518e[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:43,071 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,072 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,072 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:43,072 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:43,072 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,072 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:43,072 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:43,073 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,073 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:43,073 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:43,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:43,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:43,073 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:43,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:43,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:43,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:43,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:43,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:43,076 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:43,076 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:43,076 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:43,076 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:43,076 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:43,077 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,077 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:43,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:43,077 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:43,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:43,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:43,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:43,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:43,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,079 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:43,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,080 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,081 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,081 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,081 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,081 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:43,081 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,081 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:43,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:43,081 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:43,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:43,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:43,081 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,081 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,081 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:43,081 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:43,081 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:43,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:43,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:43,181 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:43,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:43,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:43,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:43,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:43,184 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:43,184 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:43,184 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:43,184 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:43,184 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,184 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,185 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,185 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:43,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:43,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:43,185 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,185 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:43,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:43,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,186 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:43,190 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:43,190 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:43,190 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:43,190 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:43,190 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:43,190 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:43,190 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:43,190 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:43,191 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:56:43,191 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:56:43,191 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:43,191 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:43,191 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:43,191 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,191 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:43,191 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:43,198 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,198 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:43,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:43,199 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,199 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:43,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:43,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:43,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:43,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:43,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3eeedfa9[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:43,200 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:43,200 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,206 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,206 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:43,206 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:43,206 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:43,206 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,206 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:43,207 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:43,207 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:43,207 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:43,207 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:43,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:43,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:43,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:43,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:43,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:43,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:43,208 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:43,209 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:43,209 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:43,209 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:43,209 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:43,209 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,209 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:43,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:43,209 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:43,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:43,210 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:43,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:43,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:43,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,211 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:43,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,212 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,212 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,212 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,212 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:43,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:43,212 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:43,212 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,212 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:43,212 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,212 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:43,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:43,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:43,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:43,212 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,213 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:43,213 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:43,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:43,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:43,262 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:43,262 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:43,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987463260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987463260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,263 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:43,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987463261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,263 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:43,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987463261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,263 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:43,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987463261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,313 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:43,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:43,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:43,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:43,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:43,316 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:43,316 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:43,316 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:43,316 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:43,316 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,316 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,317 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,317 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:43,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:43,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:43,317 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,317 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:43,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:43,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,317 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:43,318 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:43,318 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:43,318 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:43,318 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:43,318 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:43,318 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:43,318 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:43,318 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:43,318 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:56:43,318 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:56:43,318 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:43,318 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:43,318 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:43,318 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,318 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:43,318 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:43,320 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:43,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:43,320 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:43,320 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,320 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:43,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:43,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:43,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:43,322 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:43,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d5e0cf7[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:43,322 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,322 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,322 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,322 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:43,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,324 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:43,324 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:43,324 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:43,324 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:43,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:43,324 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:43,324 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:43,325 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:43,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:43,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:43,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:43,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:43,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:43,327 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:43,327 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:43,327 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:43,327 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:43,327 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:43,328 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,328 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:43,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:43,328 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:43,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:43,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:43,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:43,329 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:43,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,330 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:43,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,331 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,331 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:43,331 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:43,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:43,331 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:43,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:43,331 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:43,331 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:43,332 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:43,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:43,332 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:43,332 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,332 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:43,332 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,332 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,385 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/A/639ce6bba81d4d2d8ca45d8c6f41fd74 2023-04-08 20:56:43,432 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:43,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:43,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:43,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:43,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:43,435 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:43,435 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:43,435 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:43,435 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:43,435 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,435 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,436 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,436 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:43,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:43,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:43,436 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,436 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:43,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:43,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,436 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:43,437 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:43,437 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:43,437 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:43,437 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:43,437 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:43,437 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:43,437 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:43,437 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:43,437 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:56:43,437 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:56:43,437 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:43,437 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:43,437 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:43,437 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,437 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:43,437 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:43,438 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,438 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:43,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:43,438 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,438 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:43,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:43,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:43,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:43,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:43,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18005592[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:43,439 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:43,439 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,440 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,440 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:43,440 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,440 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:43,440 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:43,440 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:43,441 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,441 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:43,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:43,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:43,441 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:43,441 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:43,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:43,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:43,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:43,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:43,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:43,443 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:43,443 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:43,443 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:43,443 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:43,443 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:43,444 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,444 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,444 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:43,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:43,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:43,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:43,444 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:43,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:43,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:43,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,445 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:43,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,446 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:43,446 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:43,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:43,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:43,447 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:43,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:43,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:43,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:43,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:43,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:43,447 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,447 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:43,447 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,447 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:43,447 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,447 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,447 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,447 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,447 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,547 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:43,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:43,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:43,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:43,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:43,550 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:43,550 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:43,550 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:43,550 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:43,551 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,551 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,551 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:43,551 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:43,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:43,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,552 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,552 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:43,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:43,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,552 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:43,552 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:43,552 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:43,552 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:43,552 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:43,553 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:43,553 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:43,553 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:43,553 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:56:43,553 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:56:43,553 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:43,553 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:43,553 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:43,553 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,553 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:43,553 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:43,554 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,554 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:43,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:43,554 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,554 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:43,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:43,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:43,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:43,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:43,555 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:43,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c8efd3d[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:43,555 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,555 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,556 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:43,556 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:43,556 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:43,556 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,556 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:43,557 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:43,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:43,557 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:43,557 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:43,557 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:43,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:43,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:43,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:43,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:43,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:43,558 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:43,558 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:43,558 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:43,558 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:43,558 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:43,559 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,559 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:43,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:43,559 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:43,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:43,560 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:43,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:43,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:43,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,561 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:43,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,562 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,562 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,562 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:43,562 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:43,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:43,562 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,562 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:43,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:43,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:43,562 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:43,562 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:43,562 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:43,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:43,562 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:43,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:43,563 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,563 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,567 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:43,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987463564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,567 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:43,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987463564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,569 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:43,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987463565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,570 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:43,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987463565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,570 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:43,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987463566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,662 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:43,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:43,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:43,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:43,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:43,665 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:43,665 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:43,665 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:43,665 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:43,666 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,666 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,666 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,666 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:43,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:43,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:43,666 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,666 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:43,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:43,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,667 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:43,667 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:43,667 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:43,667 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:43,667 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:43,667 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:43,668 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:43,668 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:43,668 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:43,668 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:56:43,668 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:56:43,668 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:43,668 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:43,668 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:43,668 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,668 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:43,668 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:43,669 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:43,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:43,669 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:43,669 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,669 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:43,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:43,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:43,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:43,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54d57a92[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:43,670 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:43,670 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,671 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,671 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:43,671 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:43,671 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,671 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:43,671 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:43,672 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:43,673 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,673 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:43,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:43,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:43,673 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:43,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:43,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:43,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:43,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:43,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:43,675 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:43,675 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:43,675 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:43,675 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:43,675 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:43,676 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,676 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,676 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:43,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:43,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:43,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:43,676 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:43,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:43,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:43,678 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,679 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,679 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,679 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,679 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,679 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,679 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:43,679 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:43,679 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:43,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:43,679 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:43,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:43,679 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:43,680 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:43,680 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:43,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:43,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:43,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:43,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:43,780 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:43,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:43,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:43,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:43,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:43,783 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:43,783 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:43,783 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:43,783 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:43,783 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,783 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,784 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:43,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:43,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:43,784 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,784 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,784 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:43,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:43,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,785 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:43,785 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:43,785 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:43,785 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:43,785 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:43,785 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:43,785 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:43,785 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:43,785 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:56:43,785 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:43,785 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:56:43,785 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:43,785 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:43,785 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:43,785 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,785 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:43,785 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:43,786 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,786 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:43,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:43,786 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:43,787 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:43,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:43,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:43,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:43,788 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:43,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59e1fd4b[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:43,788 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,788 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,788 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:43,788 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,788 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:43,788 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:43,788 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:43,789 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,789 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:43,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:43,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:43,789 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:43,789 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:43,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:43,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:43,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:43,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:43,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:43,791 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:43,791 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:43,791 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:43,791 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:43,791 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:43,792 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,792 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:43,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:43,792 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:43,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:43,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:43,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:43,792 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:43,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,793 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:43,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,794 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,794 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,794 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:43,795 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:43,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:43,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:43,795 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,795 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,795 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,795 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:43,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:43,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:43,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:43,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:43,795 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:43,795 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,795 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:43,795 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:43,802 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/B/2563655cd3564c919f057633491c42c4 2023-04-08 20:56:43,895 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:43,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:43,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:43,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:43,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:43,898 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:43,898 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:43,898 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:43,898 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:43,898 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,898 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,899 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:43,899 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:43,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:43,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,899 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,900 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:43,900 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:43,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,900 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:43,900 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:43,900 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:43,900 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:43,900 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:43,900 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:43,900 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:43,900 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:43,900 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:43,901 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:56:43,901 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:56:43,901 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:43,901 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:43,901 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:43,901 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,901 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:43,901 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:43,901 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,901 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:43,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:43,902 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,902 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:43,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:43,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:43,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:43,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:43,904 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:43,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43579f20[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:43,904 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,905 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,905 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,905 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:43,905 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:43,905 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:43,905 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:43,906 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,906 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:43,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:43,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:43,906 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:43,906 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:43,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:43,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:43,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:43,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:43,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:43,907 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:43,908 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:43,908 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:43,908 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:43,908 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:43,909 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:43,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:43,909 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,909 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:43,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:43,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:43,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:43,911 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:43,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,912 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:43,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:43,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,914 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,914 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:43,914 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:43,914 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:43,914 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:43,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:43,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:43,914 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:43,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:43,914 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:43,914 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:43,914 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:43,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:43,914 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,914 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:43,914 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:43,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:43,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:44,014 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:44,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:44,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:44,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:44,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:44,017 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:44,017 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:44,017 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:44,018 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:44,018 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:44,018 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:44,019 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:44,019 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:44,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:44,019 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,019 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:44,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:44,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:44,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:44,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:44,020 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:44,020 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:44,020 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:44,020 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:44,020 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:44,020 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:44,020 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:44,020 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:44,020 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:44,020 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:56:44,020 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:56:44,020 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:44,020 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:44,020 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:44,020 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,020 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:44,021 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:44,021 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,021 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:44,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:44,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:44,022 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:44,022 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:44,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:44,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:44,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:44,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:44,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:44,023 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:44,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@285be562[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:44,023 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:44,023 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:44,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:44,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:44,024 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:44,024 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:44,024 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,024 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:44,024 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:44,024 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:44,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:44,024 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:44,025 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:44,025 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:44,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:44,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:44,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:44,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:44,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:44,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:44,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:44,027 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:44,027 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:44,027 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:44,027 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:44,027 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:44,028 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:44,029 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:44,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:44,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:44,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:44,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:44,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:44,029 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:44,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:44,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:44,029 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:44,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:44,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:44,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:44,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:44,030 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:44,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:44,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,033 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,033 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:44,033 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:44,033 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:44,033 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:44,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:44,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:44,033 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,033 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:44,033 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:44,033 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:44,033 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:44,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:44,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:44,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:44,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:44,033 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:44,033 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:44,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:44,073 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:44,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987464071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,074 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:44,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987464071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,074 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:44,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987464071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,076 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:44,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987464074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,076 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:44,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987464074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,134 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:44,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:44,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:44,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:44,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:44,136 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:44,136 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:44,136 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:44,136 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:44,136 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:44,136 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:44,137 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,137 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:44,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:44,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:44,137 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,137 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:44,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:44,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:44,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:44,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:44,138 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:44,138 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:44,138 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:44,138 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:44,138 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:44,138 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:44,138 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:44,138 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:44,139 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:44,139 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:56:44,139 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:56:44,139 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:44,139 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:44,139 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:44,139 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,139 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:44,139 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:44,140 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:44,140 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,140 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:44,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:44,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:44,140 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:44,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:44,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:44,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:44,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:44,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:44,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57a50b11[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:44,141 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:44,141 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:44,141 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:44,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:44,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:44,141 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:44,142 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:44,142 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:44,142 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,142 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:44,142 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,142 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:44,142 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:44,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:44,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:44,142 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:44,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:44,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:44,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:44,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:44,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:44,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:44,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:44,144 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:44,144 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:44,144 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:44,144 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:44,144 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:44,145 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:44,145 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:44,145 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:44,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:44,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:44,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:44,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:44,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:44,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:44,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:44,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:44,146 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:44,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:44,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:44,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:44,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,147 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:44,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:44,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,148 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:44,148 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:44,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:44,148 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:44,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:44,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:44,148 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:44,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:44,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:44,148 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:44,148 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:44,148 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:44,148 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,148 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:44,148 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:44,148 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:44,148 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:44,213 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/C/8a302cb8e9104fa7a19d469e31e2316d 2023-04-08 20:56:44,217 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/A/639ce6bba81d4d2d8ca45d8c6f41fd74 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/639ce6bba81d4d2d8ca45d8c6f41fd74 2023-04-08 20:56:44,220 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/639ce6bba81d4d2d8ca45d8c6f41fd74, entries=100, sequenceid=16, filesize=18.4 K 2023-04-08 20:56:44,220 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/B/2563655cd3564c919f057633491c42c4 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/2563655cd3564c919f057633491c42c4 2023-04-08 20:56:44,223 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/2563655cd3564c919f057633491c42c4, entries=100, sequenceid=16, filesize=9.3 K 2023-04-08 20:56:44,224 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/C/8a302cb8e9104fa7a19d469e31e2316d as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/8a302cb8e9104fa7a19d469e31e2316d 2023-04-08 20:56:44,228 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/8a302cb8e9104fa7a19d469e31e2316d, entries=100, sequenceid=16, filesize=9.3 K 2023-04-08 20:56:44,229 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for eb4c5ed1339ded9a0ba1af8980e65ffc in 1289ms, sequenceid=16, compaction requested=false 2023-04-08 20:56:44,229 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for eb4c5ed1339ded9a0ba1af8980e65ffc: 2023-04-08 20:56:44,248 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:44,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:44,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:44,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:44,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:44,251 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:44,251 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:44,251 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:44,251 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:44,252 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:44,252 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:44,252 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,252 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:44,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:44,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:44,252 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:44,252 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:44,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:44,253 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:44,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:44,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:44,253 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:44,253 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:44,253 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:44,253 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:44,253 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:44,253 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:44,254 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:44,254 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:44,254 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:44,254 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:56:44,254 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing eb4c5ed1339ded9a0ba1af8980e65ffc 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-08 20:56:44,254 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=A 2023-04-08 20:56:44,254 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:44,254 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=B 2023-04-08 20:56:44,254 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:44,254 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=C 2023-04-08 20:56:44,254 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:44,265 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408deb0b5b44fad4609a1b0b39d126de98d to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408deb0b5b44fad4609a1b0b39d126de98d 2023-04-08 20:56:44,670 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/A/4a3dca94131c4a1d8dcbd042d9fdebe7 2023-04-08 20:56:45,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:56:45,079 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:56:45,081 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/B/44a2955b55dc48a08a27f3dfe2f32859 2023-04-08 20:56:45,096 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:45,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987465089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,096 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:45,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987465090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,099 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:45,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987465092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,099 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:45,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987465094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,099 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:45,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987465095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,198 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:45,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987465197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,199 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:45,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987465197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,203 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:45,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987465200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,203 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:45,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987465201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,204 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:45,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987465201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,403 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:45,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987465399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,403 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:45,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987465400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,409 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:45,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987465404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,410 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:45,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987465404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:45,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987465405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,509 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/C/9a5f54890148437885355dc1c3c9b2ac 2023-04-08 20:56:45,513 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/A/4a3dca94131c4a1d8dcbd042d9fdebe7 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/4a3dca94131c4a1d8dcbd042d9fdebe7 2023-04-08 20:56:45,516 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/4a3dca94131c4a1d8dcbd042d9fdebe7, entries=150, sequenceid=36, filesize=25.1 K 2023-04-08 20:56:45,517 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/B/44a2955b55dc48a08a27f3dfe2f32859 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/44a2955b55dc48a08a27f3dfe2f32859 2023-04-08 20:56:45,520 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/44a2955b55dc48a08a27f3dfe2f32859, entries=150, sequenceid=36, filesize=11.6 K 2023-04-08 20:56:45,521 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/C/9a5f54890148437885355dc1c3c9b2ac as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/9a5f54890148437885355dc1c3c9b2ac 2023-04-08 20:56:45,524 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/9a5f54890148437885355dc1c3c9b2ac, entries=150, sequenceid=36, filesize=11.6 K 2023-04-08 20:56:45,524 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for eb4c5ed1339ded9a0ba1af8980e65ffc in 1270ms, sequenceid=36, compaction requested=false 2023-04-08 20:56:45,524 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for eb4c5ed1339ded9a0ba1af8980e65ffc: 2023-04-08 20:56:45,524 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:45,524 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:45,524 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:45,525 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,525 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:45,525 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:45,525 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,525 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:45,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:45,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:45,526 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:45,526 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:45,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:45,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:45,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:45,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:45,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:45,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f8bf0b[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:45,527 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:45,527 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:45,527 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:45,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:45,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:45,528 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:45,528 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:45,528 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,528 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:45,528 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:45,528 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,528 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:45,528 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:45,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:45,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:45,528 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:45,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:45,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:45,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:45,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:45,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:45,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:45,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:45,530 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:45,530 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:45,530 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:45,530 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:45,530 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:45,531 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:45,531 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:45,531 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:45,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:45,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:45,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:45,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:45,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:45,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:45,532 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:45,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:45,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:45,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:45,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:45,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:45,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,533 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,534 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,534 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:45,534 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:45,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:45,534 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:45,534 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:45,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:45,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:45,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:45,534 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:45,534 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,534 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:45,534 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:45,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:45,534 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:45,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:45,534 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:45,534 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:45,600 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-08 20:56:45,634 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:45,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:45,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:45,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:45,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:45,637 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:45,637 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:45,637 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:45,637 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:45,637 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:45,637 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:45,638 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,638 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:45,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:45,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:45,638 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,638 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:45,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:45,639 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:45,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:45,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:45,639 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:45,639 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:45,639 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:45,639 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:45,639 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:45,640 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:45,640 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:45,640 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:45,640 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:45,640 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:56:45,640 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing eb4c5ed1339ded9a0ba1af8980e65ffc 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-08 20:56:45,640 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=A 2023-04-08 20:56:45,640 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:45,640 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=B 2023-04-08 20:56:45,640 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:45,640 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=C 2023-04-08 20:56:45,640 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:45,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:56:45,710 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:56:45,732 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:45,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987465728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,736 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:45,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987465729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,736 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:45,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987465730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,739 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:45,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987465731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,739 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:45,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987465732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:45,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987465833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,838 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:45,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987465837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,841 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:45,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987465837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:45,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987465840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:45,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:45,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987465840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:46,039 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:46,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987466038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:46,042 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:46,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987466039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:46,045 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:46,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987466041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:46,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:46,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987466045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:46,048 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:46,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987466045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:46,068 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304085fa758a6331e4c5199027394041fa4f3 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304085fa758a6331e4c5199027394041fa4f3 2023-04-08 20:56:46,343 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:46,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987466341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:46,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:46,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987466344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:46,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:46,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987466348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:46,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:46,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987466349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:46,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:46,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987466349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:46,472 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=48, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/A/0989381648ff4abfabb2800bdfc467d5 2023-04-08 20:56:46,848 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:46,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987466845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:46,855 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:46,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987466852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:46,856 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:46,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987466853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:46,860 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:46,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987466858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:46,860 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:46,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987466858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:46,884 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/B/965b8359c91e4630ba7366f7c2032b2f 2023-04-08 20:56:47,292 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/C/7fc987c4d213417bbb894481dcffe159 2023-04-08 20:56:47,296 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/A/0989381648ff4abfabb2800bdfc467d5 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/0989381648ff4abfabb2800bdfc467d5 2023-04-08 20:56:47,299 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/0989381648ff4abfabb2800bdfc467d5, entries=150, sequenceid=48, filesize=25.1 K 2023-04-08 20:56:47,300 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/B/965b8359c91e4630ba7366f7c2032b2f as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/965b8359c91e4630ba7366f7c2032b2f 2023-04-08 20:56:47,303 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/965b8359c91e4630ba7366f7c2032b2f, entries=150, sequenceid=48, filesize=11.6 K 2023-04-08 20:56:47,304 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/C/7fc987c4d213417bbb894481dcffe159 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/7fc987c4d213417bbb894481dcffe159 2023-04-08 20:56:47,307 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/7fc987c4d213417bbb894481dcffe159, entries=150, sequenceid=48, filesize=11.6 K 2023-04-08 20:56:47,307 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for eb4c5ed1339ded9a0ba1af8980e65ffc in 1667ms, sequenceid=48, compaction requested=true 2023-04-08 20:56:47,308 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for eb4c5ed1339ded9a0ba1af8980e65ffc: 2023-04-08 20:56:47,308 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:47,308 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:47,308 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:47,308 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:47,308 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:47,308 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:47,309 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:47,309 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:47,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:47,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:47,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:47,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:47,309 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:47,309 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:47,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:47,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:47,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:47,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:47,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:47,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ea99e8b[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:47,310 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:47,310 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:47,311 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:47,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:47,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:47,311 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:47,311 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:47,311 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:47,311 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:47,311 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:47,313 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:47,313 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:47,313 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:47,313 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:47,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:47,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:47,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:47,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:47,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:47,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:47,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:47,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:47,315 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:47,315 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:47,315 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:47,315 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:47,315 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:47,316 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:47,316 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:47,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:47,316 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:47,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:47,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:47,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:47,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:47,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:47,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:47,316 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:47,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:47,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:47,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:47,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:47,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:47,318 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:47,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:47,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:47,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:47,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:47,321 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:47,321 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:47,321 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:47,321 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:47,321 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:47,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:47,322 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:47,322 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:47,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:47,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:47,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:47,322 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:47,322 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:47,322 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:47,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:47,322 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:47,322 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:47,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:47,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:47,422 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:47,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:47,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:47,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:47,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:47,425 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:47,425 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:47,425 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:47,425 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:47,425 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:47,425 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:47,426 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:47,426 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:47,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:47,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:47,426 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:47,426 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:47,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:47,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:47,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:47,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:47,427 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:47,427 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:47,427 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:47,427 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:47,427 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:47,427 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:47,427 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:47,427 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:47,427 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:47,427 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:56:47,427 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing eb4c5ed1339ded9a0ba1af8980e65ffc 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-08 20:56:47,427 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=A 2023-04-08 20:56:47,427 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:47,427 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=B 2023-04-08 20:56:47,427 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:47,427 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=C 2023-04-08 20:56:47,427 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:47,838 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304087c0c1228de324ced87aff08724079fa1 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304087c0c1228de324ced87aff08724079fa1 2023-04-08 20:56:47,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:56:47,861 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:56:47,907 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:47,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987467897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:47,907 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:47,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987467902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:47,912 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:47,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987467906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:47,912 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:47,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987467907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:47,913 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:47,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987467907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:48,013 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:48,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987468008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:48,016 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:48,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987468008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:48,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:48,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987468013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:48,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:48,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987468013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:48,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:48,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987468013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:48,219 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:48,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987468214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:48,219 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:48,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987468217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:48,220 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:48,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987468218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:48,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:48,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987468218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:48,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:48,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987468219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:48,243 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=68, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/A/f1618df8c11b4e0bad0800a796be3418 2023-04-08 20:56:48,525 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:48,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987468520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:48,525 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:48,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987468520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:48,525 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:48,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987468520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:48,525 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:48,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987468523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:48,526 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:48,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987468524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:48,654 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/B/bbf254924bdf4b49a8db43108a5eee2f 2023-04-08 20:56:49,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:49,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987469027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:49,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:49,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987469028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:49,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:49,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987469028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:49,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:49,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987469028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:49,034 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:49,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987469030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:49,062 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/C/0a688d4ae27a4c0185ab6105c5893415 2023-04-08 20:56:49,066 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/A/f1618df8c11b4e0bad0800a796be3418 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/f1618df8c11b4e0bad0800a796be3418 2023-04-08 20:56:49,069 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/f1618df8c11b4e0bad0800a796be3418, entries=150, sequenceid=68, filesize=25.1 K 2023-04-08 20:56:49,071 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/B/bbf254924bdf4b49a8db43108a5eee2f as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/bbf254924bdf4b49a8db43108a5eee2f 2023-04-08 20:56:49,074 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/bbf254924bdf4b49a8db43108a5eee2f, entries=150, sequenceid=68, filesize=11.6 K 2023-04-08 20:56:49,075 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/C/0a688d4ae27a4c0185ab6105c5893415 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/0a688d4ae27a4c0185ab6105c5893415 2023-04-08 20:56:49,077 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/0a688d4ae27a4c0185ab6105c5893415, entries=150, sequenceid=68, filesize=11.6 K 2023-04-08 20:56:49,078 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for eb4c5ed1339ded9a0ba1af8980e65ffc in 1651ms, sequenceid=68, compaction requested=true 2023-04-08 20:56:49,078 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for eb4c5ed1339ded9a0ba1af8980e65ffc: 2023-04-08 20:56:49,078 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:49,078 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:49,078 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:49,078 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:49,078 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:49,078 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:49,079 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:49,079 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:49,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:49,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:49,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:49,079 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:49,079 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:49,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:49,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:49,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:49,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:49,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:49,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:49,080 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:49,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@92e14c9[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:49,080 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:49,081 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:49,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:49,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:49,081 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:49,081 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:49,081 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:49,081 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:49,081 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:49,082 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:49,082 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:49,082 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:49,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:49,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:49,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:49,082 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:49,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:49,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:49,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:49,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:49,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:49,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:49,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:49,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:49,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:49,085 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:49,085 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:49,085 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:49,085 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:49,085 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:49,086 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:49,086 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:49,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:49,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:49,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:49,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:49,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:49,086 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:49,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:49,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:49,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:49,086 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:49,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:49,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:49,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:49,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:49,087 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:49,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:49,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:49,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:49,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:49,089 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:49,089 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:49,089 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:49,089 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:49,089 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:49,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:49,089 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:49,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:49,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:49,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:49,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:49,089 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:49,089 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:49,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:49,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:49,089 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:49,089 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:49,089 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:49,089 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:49,190 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:49,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:49,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:49,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:49,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:49,193 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:49,193 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:49,193 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:49,193 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:49,193 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:49,193 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:49,194 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:49,194 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:49,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:49,194 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:49,194 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:49,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:49,195 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:49,195 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:49,195 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:49,195 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:49,195 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:49,195 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:49,195 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:49,195 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:49,195 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:49,195 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:56:49,195 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing eb4c5ed1339ded9a0ba1af8980e65ffc 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-08 20:56:49,196 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=A 2023-04-08 20:56:49,196 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:49,196 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=B 2023-04-08 20:56:49,196 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:49,196 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=C 2023-04-08 20:56:49,196 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:49,608 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304089e69e58721a54ffb8d5765e0e791fe6d to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304089e69e58721a54ffb8d5765e0e791fe6d 2023-04-08 20:56:49,613 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=81, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/A/47048196d3464a46a6c44457bdbde2ec 2023-04-08 20:56:50,022 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/B/a44f51f2b80c4c898cfe5af94538c04c 2023-04-08 20:56:50,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:56:50,042 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:56:50,070 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:50,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987470062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,070 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:50,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987470062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,074 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:50,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987470062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,074 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:50,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987470063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,080 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:50,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987470070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,173 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:50,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987470171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,173 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:50,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987470171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:50,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987470175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:50,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987470175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:50,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987470181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,374 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:50,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987470374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,380 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:50,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987470374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:50,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987470380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:50,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987470381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,387 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:50,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987470385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,430 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/C/480ebefa348d459cb4b826f360134ae5 2023-04-08 20:56:50,433 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/A/47048196d3464a46a6c44457bdbde2ec as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/47048196d3464a46a6c44457bdbde2ec 2023-04-08 20:56:50,436 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/47048196d3464a46a6c44457bdbde2ec, entries=150, sequenceid=81, filesize=25.1 K 2023-04-08 20:56:50,437 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/B/a44f51f2b80c4c898cfe5af94538c04c as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/a44f51f2b80c4c898cfe5af94538c04c 2023-04-08 20:56:50,440 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/a44f51f2b80c4c898cfe5af94538c04c, entries=150, sequenceid=81, filesize=11.6 K 2023-04-08 20:56:50,440 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/C/480ebefa348d459cb4b826f360134ae5 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/480ebefa348d459cb4b826f360134ae5 2023-04-08 20:56:50,443 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/480ebefa348d459cb4b826f360134ae5, entries=150, sequenceid=81, filesize=11.6 K 2023-04-08 20:56:50,444 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for eb4c5ed1339ded9a0ba1af8980e65ffc in 1249ms, sequenceid=81, compaction requested=true 2023-04-08 20:56:50,444 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for eb4c5ed1339ded9a0ba1af8980e65ffc: 2023-04-08 20:56:50,444 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:50,444 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:50,444 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:50,444 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,444 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:50,444 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:50,445 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,445 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:50,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:50,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:50,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:50,446 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:50,446 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:50,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:50,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:50,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:50,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:50,447 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:50,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@762c7b7e[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:50,447 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:50,447 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:50,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:50,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:50,447 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:50,447 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,447 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:50,447 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:50,447 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:50,448 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:50,448 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:50,448 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,448 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:50,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:50,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:50,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:50,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:50,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:50,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:50,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:50,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:50,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:50,450 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:50,450 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:50,450 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:50,450 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:50,450 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:50,451 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:50,451 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:50,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:50,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:50,451 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:50,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:50,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:50,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:50,451 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:50,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:50,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:50,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:50,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:50,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:50,452 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:50,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:50,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:50,454 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:50,454 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:50,454 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:50,454 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:50,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:50,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:50,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:50,454 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:50,454 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:50,454 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,454 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:50,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:50,454 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:50,454 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:50,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:50,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:50,454 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:50,454 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:50,554 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:50,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:50,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:50,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:50,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:50,557 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:50,557 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:50,557 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:50,557 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:50,557 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:50,557 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:50,558 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:50,558 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:50,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:50,558 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,558 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:50,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:50,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:50,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:50,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:50,559 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:50,559 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:50,559 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:50,559 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:50,559 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:50,559 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:50,559 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:50,559 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:50,559 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:50,559 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:56:50,560 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing eb4c5ed1339ded9a0ba1af8980e65ffc 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-08 20:56:50,560 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=A 2023-04-08 20:56:50,560 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:50,560 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=B 2023-04-08 20:56:50,560 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:50,560 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=C 2023-04-08 20:56:50,560 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:50,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:56:50,679 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:56:50,698 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:50,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987470689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,704 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:50,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987470693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,706 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:50,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987470696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,706 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:50,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987470697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,706 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:50,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987470698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:50,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987470805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,811 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:50,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987470806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,811 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:50,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987470807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,811 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:50,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987470807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:50,975 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408f3ac260347a94d6aa6b767c933916a87 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408f3ac260347a94d6aa6b767c933916a87 2023-04-08 20:56:50,980 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=100, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/A/e8853eb9f0f64918a56f5b6f326d57a6 2023-04-08 20:56:51,024 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:51,024 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:51,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987471012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987471012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,024 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:51,024 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:51,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987471013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987471013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,206 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:51,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987471201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,331 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:51,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987471327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,331 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:51,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987471327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,331 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:51,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987471327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:51,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987471327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,391 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/B/fdd43aa340584c03a1ecdc811a9c63fb 2023-04-08 20:56:51,800 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/C/818a54bbaf484dca92c437bc61db2831 2023-04-08 20:56:51,803 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/A/e8853eb9f0f64918a56f5b6f326d57a6 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/e8853eb9f0f64918a56f5b6f326d57a6 2023-04-08 20:56:51,806 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/e8853eb9f0f64918a56f5b6f326d57a6, entries=150, sequenceid=100, filesize=25.1 K 2023-04-08 20:56:51,807 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/B/fdd43aa340584c03a1ecdc811a9c63fb as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/fdd43aa340584c03a1ecdc811a9c63fb 2023-04-08 20:56:51,810 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/fdd43aa340584c03a1ecdc811a9c63fb, entries=150, sequenceid=100, filesize=11.6 K 2023-04-08 20:56:51,811 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/C/818a54bbaf484dca92c437bc61db2831 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/818a54bbaf484dca92c437bc61db2831 2023-04-08 20:56:51,814 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/818a54bbaf484dca92c437bc61db2831, entries=150, sequenceid=100, filesize=11.6 K 2023-04-08 20:56:51,814 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for eb4c5ed1339ded9a0ba1af8980e65ffc in 1254ms, sequenceid=100, compaction requested=true 2023-04-08 20:56:51,814 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for eb4c5ed1339ded9a0ba1af8980e65ffc: 2023-04-08 20:56:51,814 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:51,814 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:51,815 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:51,815 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,815 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:51,815 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:51,815 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,815 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:51,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:51,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:51,816 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:51,816 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:51,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:51,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:51,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:51,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:51,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:51,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32689257[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:51,817 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:51,817 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:51,817 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:51,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:51,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:51,817 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:51,818 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:51,818 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:51,818 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,818 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:51,818 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,818 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:51,818 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:51,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:51,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:51,818 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:51,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:51,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:51,820 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:51,820 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:51,820 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:51,820 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:51,820 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:51,821 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:51,821 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:51,821 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:51,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:51,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:51,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:51,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:51,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:51,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:51,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:51,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:51,822 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:51,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:51,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:51,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:51,824 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:51,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:51,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,826 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:51,826 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:51,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:51,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:51,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:51,826 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:51,827 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:51,827 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:51,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:51,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:51,826 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:51,826 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:51,827 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:51,827 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,827 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:51,827 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:51,827 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:51,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:56:51,836 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing eb4c5ed1339ded9a0ba1af8980e65ffc 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-08 20:56:51,837 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=A 2023-04-08 20:56:51,837 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:51,837 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=B 2023-04-08 20:56:51,837 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:51,837 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=C 2023-04-08 20:56:51,837 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:51,848 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304080673231aedeb4fbc8ac73a9cab6aaeaf to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304080673231aedeb4fbc8ac73a9cab6aaeaf 2023-04-08 20:56:51,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:51,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987471861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:51,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987471862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,876 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:51,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987471869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,876 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:51,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987471869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,927 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:51,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:51,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:51,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:51,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:51,929 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:51,929 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:51,930 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:51,930 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:51,930 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:51,930 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:51,931 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,931 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:51,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:51,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:51,931 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,931 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:51,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:51,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:51,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:51,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:51,931 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:51,932 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:51,932 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:51,932 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:51,932 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:51,932 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:51,932 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:51,932 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:51,932 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:51,932 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:56:51,932 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:56:51,933 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:51,933 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:51,933 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:51,933 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,933 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:51,933 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:51,934 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,934 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:51,934 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:51,934 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:51,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:51,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:51,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70f57[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:51,935 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:51,935 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:51,936 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:51,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:51,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:51,936 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:51,936 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:51,936 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:51,936 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,936 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:51,937 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,937 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:51,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:51,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:51,937 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:51,938 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:51,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:51,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:51,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:51,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:51,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:51,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:51,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:51,940 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:51,940 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:51,940 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:51,940 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:51,940 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:51,956 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:51,956 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:51,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:51,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:51,956 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:51,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:51,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:51,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:51,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:51,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:51,957 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:51,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:51,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:51,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:51,958 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:51,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:51,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:51,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,960 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,960 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:51,960 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:51,960 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:51,960 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:51,960 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:51,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:51,960 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:51,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:51,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:51,960 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:51,960 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:51,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:51,960 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:51,960 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:51,960 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:51,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:51,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:51,976 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:51,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987471970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,976 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:51,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987471970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:51,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987471977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:51,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:51,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987471977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,060 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:52,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:52,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:52,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:52,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:52,063 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:52,063 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:52,063 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:52,063 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:52,063 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,063 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,064 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:52,064 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:52,064 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,064 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:52,065 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:52,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,065 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:52,065 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:52,065 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:52,065 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:52,065 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:52,065 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:52,065 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:52,066 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:52,066 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:52,066 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:56:52,066 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:56:52,066 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:52,066 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:52,066 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:52,066 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,066 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:52,066 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:52,067 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,067 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:52,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:52,067 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,067 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:52,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:52,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:52,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:52,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:52,068 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:52,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7976dbc2[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:52,068 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,069 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,069 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:52,069 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:52,069 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:52,069 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,069 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:52,069 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,069 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:52,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:52,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:52,069 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:52,070 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:52,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:52,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:52,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:52,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:52,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:52,071 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:52,071 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:52,071 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:52,072 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:52,072 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:52,072 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,072 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:52,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:52,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,072 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:52,073 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:52,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:52,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:52,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:52,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,074 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:52,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,075 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,075 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,075 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,075 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:52,075 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:52,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:52,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:52,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:52,075 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:52,075 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:52,075 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:52,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:52,075 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:52,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:52,075 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:52,075 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,076 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,176 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:52,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:52,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:52,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:52,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:52,179 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:52,179 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:52,179 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:52,179 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:52,179 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,179 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,180 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:52,180 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:52,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:52,180 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,180 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:52,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:52,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,181 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:52,181 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:52,181 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:52,181 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:52,181 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:52,181 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:52,182 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:52,182 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:52,182 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:52,182 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:52,182 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:56:52,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987472177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,182 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:56:52,182 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:52,182 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:52,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987472177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,183 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:52,183 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:52,183 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,184 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:52,184 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:52,186 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:52,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987472180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,186 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:52,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987472181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,190 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,190 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:52,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:52,190 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:52,190 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:52,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:52,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:52,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:52,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c94b819[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:52,192 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:52,192 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,197 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,198 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:52,198 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,198 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:52,198 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:52,198 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:52,199 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:52,199 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,199 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:52,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:52,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:52,199 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:52,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:52,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:52,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:52,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:52,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:52,201 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:52,201 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:52,201 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:52,201 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:52,201 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:52,202 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,202 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:52,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:52,202 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:52,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:52,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:52,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:52,204 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:52,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,206 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:52,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,211 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:52,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987472209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,214 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,214 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,214 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:52,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:52,214 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:52,214 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:52,215 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:52,214 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,215 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:52,215 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:52,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:52,215 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,215 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:52,215 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:52,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:52,253 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=115, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/A/419be3f2c9224c4bbf0c2ed1d9a5e7cf 2023-04-08 20:56:52,315 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:52,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:52,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:52,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:52,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:52,317 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:52,317 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:52,317 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:52,317 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:52,318 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,318 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,318 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,318 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:52,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:52,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:52,319 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,319 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:52,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:52,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,319 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:52,319 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:52,319 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:52,320 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:52,320 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:52,320 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:52,320 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:52,320 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:52,320 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:52,320 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:56:52,320 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:56:52,320 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:52,320 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:52,320 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:52,320 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,320 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:52,320 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:52,321 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,321 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:52,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:52,322 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,322 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:52,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:52,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:52,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:52,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:52,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a1875f6[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:52,323 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:52,323 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,323 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,323 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:52,323 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:52,323 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:52,326 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,326 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:52,326 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:52,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:52,327 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:52,327 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:52,327 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:52,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:52,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:52,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:52,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:52,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:52,328 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:52,328 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:52,328 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:52,328 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:52,328 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:52,329 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:52,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:52,329 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,329 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:52,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:52,329 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:52,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:52,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:52,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,330 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:52,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,333 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,333 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,333 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,333 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,333 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,333 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,333 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:52,334 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:52,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:52,334 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:52,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:52,334 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:52,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:52,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:52,334 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:52,334 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:52,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:52,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:52,434 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:52,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:52,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:52,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:52,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:52,437 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:52,437 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:52,437 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:52,437 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:52,438 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,438 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,438 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:52,438 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:52,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:52,439 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,439 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:52,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:52,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:52,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:52,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:52,440 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:52,440 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:52,440 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:52,440 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:52,440 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:52,440 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:52,440 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:56:52,440 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:56:52,440 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:52,440 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:52,440 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:52,440 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,440 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:52,440 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:52,441 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,441 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:52,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:52,441 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,441 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:52,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:52,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:52,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:52,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:52,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@225dac55[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:52,442 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:52,442 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,443 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,443 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:52,443 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,443 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:52,443 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:52,443 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:52,444 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,444 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:52,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:52,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:52,444 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:52,444 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:52,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:52,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:52,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:52,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:52,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:52,446 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:52,446 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:52,446 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:52,446 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:52,446 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:52,447 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,447 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:52,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:52,447 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:52,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:52,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:52,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:52,448 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:52,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,449 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:52,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,450 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,450 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,450 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,459 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,459 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,459 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,459 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,459 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:52,459 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:52,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:52,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:52,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:52,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:52,460 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:52,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:52,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:52,461 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:52,461 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:52,487 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:52,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987472484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,490 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:52,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987472485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,491 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:52,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987472487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,491 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:52,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987472487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,561 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:52,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:52,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:52,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:52,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:52,563 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:52,563 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:52,564 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:52,564 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:52,564 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,564 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,564 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:52,565 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:52,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:52,565 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,565 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:52,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:52,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,565 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:52,566 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:52,566 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:52,566 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:52,566 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:52,566 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:52,566 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:52,566 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:52,566 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:56:52,566 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:56:52,566 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:52,566 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:52,566 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:52,566 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,566 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:52,566 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:52,567 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,567 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:52,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:52,567 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,567 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:52,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:52,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:52,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:52,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:52,569 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:52,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dbadb17[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:52,569 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,569 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,569 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:52,570 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:52,570 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:52,570 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,570 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:52,570 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,570 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:52,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:52,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:52,570 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:52,570 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:52,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:52,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:52,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:52,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:52,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:52,572 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:52,572 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:52,572 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:52,572 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:52,572 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:52,573 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,573 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:52,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:52,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,573 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:52,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:52,574 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:52,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:52,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:52,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,574 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:52,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,576 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,576 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,576 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,576 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:52,576 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:52,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:52,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:52,576 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,576 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:52,576 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:52,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:52,576 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:52,576 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:52,576 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,576 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:52,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:52,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:52,664 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/B/58c388679c3b46d69c5df366bec67c7f 2023-04-08 20:56:52,676 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:52,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:52,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:52,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:52,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:52,680 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:52,680 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:52,680 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:52,680 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:52,681 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,681 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,682 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,682 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:52,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:52,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:52,682 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,682 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:52,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,683 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:52,683 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:52,683 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:52,683 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:52,683 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:52,683 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:52,683 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:52,683 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:52,683 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:52,683 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:56:52,683 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:56:52,684 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:52,684 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:52,684 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:52,684 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,684 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:52,684 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:52,684 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,684 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:52,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:52,686 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,686 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:52,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:52,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:52,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:52,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:52,688 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:52,688 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24a7e2c9[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:52,689 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,689 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:52,689 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:52,689 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:52,689 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,689 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:52,690 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:52,690 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:52,690 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,690 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:52,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:52,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:52,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:52,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:52,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:52,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:52,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:52,692 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:52,692 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:52,692 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:52,693 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:52,693 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:52,693 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,693 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,693 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:52,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:52,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:52,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:52,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:52,694 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:52,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:52,695 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:52,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,699 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,699 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:52,699 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,699 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,699 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:52,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:52,699 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:52,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:52,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:52,699 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,700 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,700 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,699 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:52,700 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:52,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:52,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:52,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:52,700 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:52,800 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:52,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:52,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:52,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:52,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:52,803 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:52,803 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:52,803 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:52,803 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:52,803 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,803 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,804 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:52,804 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:52,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:52,804 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,804 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:52,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:52,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,805 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:52,805 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:52,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:52,805 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:52,805 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:52,805 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:52,805 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:52,805 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:52,805 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:52,805 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:56:52,805 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:56:52,805 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:52,805 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:52,805 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:52,805 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,806 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:52,806 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:52,806 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,806 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:52,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:52,806 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,807 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:52,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:52,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:52,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:52,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:52,809 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:52,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d49846e[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:52,809 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,809 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,809 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:52,810 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:52,810 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:52,810 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:52,810 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,810 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:52,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:52,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:52,810 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:52,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:52,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:52,811 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,811 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:52,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:52,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:52,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:52,812 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:52,812 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:52,812 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:52,812 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:52,812 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:52,813 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,813 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,813 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:52,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:52,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:52,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:52,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:52,814 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:52,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:52,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,815 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:52,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,816 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,816 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,816 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,816 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,816 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,816 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,816 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:52,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:52,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:52,816 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:52,816 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:52,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:52,816 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:52,816 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:52,816 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:52,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:52,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:52,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:52,916 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:52,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:52,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:52,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:52,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:52,919 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:52,919 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:52,919 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:52,919 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:52,920 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,920 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,920 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,920 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:52,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:52,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:52,921 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,921 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:52,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:52,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:52,921 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:52,921 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:52,922 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:52,922 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:52,922 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:52,922 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:52,922 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:52,922 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:52,922 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:56:52,922 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:56:52,922 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:52,922 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:52,922 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:52,922 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,922 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:52,922 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:52,923 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,923 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:52,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:52,923 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,923 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:52,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:52,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:52,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:52,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:52,924 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:52,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3239a096[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:52,924 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,925 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,925 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:52,925 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:52,925 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:52,925 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,925 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:52,925 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,926 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:52,926 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:52,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:52,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:52,926 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:52,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:52,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:52,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:52,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:52,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:52,928 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:52,928 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:52,928 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:52,928 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:52,929 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:52,929 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,929 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:52,929 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:52,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:52,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:52,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:52,930 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:52,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:52,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,931 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:52,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:52,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,933 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:52,933 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:52,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:52,933 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,933 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:52,933 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:52,933 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,933 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:52,933 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,933 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:52,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:52,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:52,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:52,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:52,933 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:52,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:52,934 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:52,934 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:52,997 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:52,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987472990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:52,997 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:52,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987472992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:53,001 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:53,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987472995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:53,001 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:53,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987472996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:53,034 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:53,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:53,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:53,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:53,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:53,036 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:53,036 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:53,036 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:53,036 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:53,037 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:53,037 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:53,037 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:53,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:53,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:53,037 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:53,038 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:53,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:53,038 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:53,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:53,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:53,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:53,038 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:53,038 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:53,038 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:53,038 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:53,039 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:53,039 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:53,039 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:53,039 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:53,039 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:53,039 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:56:53,039 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:56:53,039 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:53,039 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:53,039 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:53,039 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:53,039 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:53,039 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:53,040 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:53,040 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:53,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:53,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:53,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:53,040 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:53,040 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:53,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:53,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:53,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:53,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:53,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:53,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:53,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@960b7a5[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:53,041 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:53,041 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:53,042 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:53,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:53,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:53,042 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:53,042 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:53,042 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:53,042 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:53,042 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:53,043 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:53,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:53,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:53,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:53,043 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:53,043 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:53,043 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:53,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:53,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:53,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:53,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:53,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:53,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:53,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:53,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:53,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:53,045 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:53,045 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:53,045 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:53,045 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:53,045 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:53,046 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:53,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:53,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:53,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:53,046 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:53,046 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:53,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:53,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:53,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:53,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:53,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:53,047 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:53,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:53,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:53,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:53,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:53,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:53,048 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:53,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:53,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:53,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:53,049 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:53,049 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:53,049 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:53,049 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:53,049 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:53,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:53,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:53,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:53,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:53,049 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:53,049 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:53,049 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:53,049 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:53,049 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:53,049 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:53,049 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:53,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:53,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:53,074 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/C/8b3b2580a6d4455182a60a9a5d6dc980 2023-04-08 20:56:53,077 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/A/419be3f2c9224c4bbf0c2ed1d9a5e7cf as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/419be3f2c9224c4bbf0c2ed1d9a5e7cf 2023-04-08 20:56:53,081 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/419be3f2c9224c4bbf0c2ed1d9a5e7cf, entries=250, sequenceid=115, filesize=38.7 K 2023-04-08 20:56:53,082 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/B/58c388679c3b46d69c5df366bec67c7f as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/58c388679c3b46d69c5df366bec67c7f 2023-04-08 20:56:53,084 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/58c388679c3b46d69c5df366bec67c7f, entries=150, sequenceid=115, filesize=11.6 K 2023-04-08 20:56:53,085 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/C/8b3b2580a6d4455182a60a9a5d6dc980 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/8b3b2580a6d4455182a60a9a5d6dc980 2023-04-08 20:56:53,088 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/8b3b2580a6d4455182a60a9a5d6dc980, entries=150, sequenceid=115, filesize=11.6 K 2023-04-08 20:56:53,088 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for eb4c5ed1339ded9a0ba1af8980e65ffc in 1252ms, sequenceid=115, compaction requested=true 2023-04-08 20:56:53,089 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for eb4c5ed1339ded9a0ba1af8980e65ffc: 2023-04-08 20:56:53,089 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:56:53,089 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-08 20:56:53,089 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-08 20:56:53,089 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:56:53,089 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:56:53,090 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 187158 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-08 20:56:53,090 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): eb4c5ed1339ded9a0ba1af8980e65ffc/A is initiating minor compaction (all files) 2023-04-08 20:56:53,090 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 80858 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-08 20:56:53,090 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): eb4c5ed1339ded9a0ba1af8980e65ffc/B is initiating minor compaction (all files) 2023-04-08 20:56:53,090 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of eb4c5ed1339ded9a0ba1af8980e65ffc/A in TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:53,090 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of eb4c5ed1339ded9a0ba1af8980e65ffc/B in TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:53,090 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/639ce6bba81d4d2d8ca45d8c6f41fd74, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/4a3dca94131c4a1d8dcbd042d9fdebe7, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/0989381648ff4abfabb2800bdfc467d5, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/f1618df8c11b4e0bad0800a796be3418, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/47048196d3464a46a6c44457bdbde2ec, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/e8853eb9f0f64918a56f5b6f326d57a6, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/419be3f2c9224c4bbf0c2ed1d9a5e7cf] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp, totalSize=182.8 K 2023-04-08 20:56:53,090 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/2563655cd3564c919f057633491c42c4, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/44a2955b55dc48a08a27f3dfe2f32859, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/965b8359c91e4630ba7366f7c2032b2f, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/bbf254924bdf4b49a8db43108a5eee2f, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/a44f51f2b80c4c898cfe5af94538c04c, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/fdd43aa340584c03a1ecdc811a9c63fb, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/58c388679c3b46d69c5df366bec67c7f] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp, totalSize=79.0 K 2023-04-08 20:56:53,091 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 639ce6bba81d4d2d8ca45d8c6f41fd74, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680987402935 2023-04-08 20:56:53,092 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 2563655cd3564c919f057633491c42c4, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680987402935 2023-04-08 20:56:53,092 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 4a3dca94131c4a1d8dcbd042d9fdebe7, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680987402951 2023-04-08 20:56:53,093 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 44a2955b55dc48a08a27f3dfe2f32859, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680987402951 2023-04-08 20:56:53,093 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 0989381648ff4abfabb2800bdfc467d5, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680987405091 2023-04-08 20:56:53,093 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 965b8359c91e4630ba7366f7c2032b2f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680987405091 2023-04-08 20:56:53,094 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting f1618df8c11b4e0bad0800a796be3418, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1680987405729 2023-04-08 20:56:53,094 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting bbf254924bdf4b49a8db43108a5eee2f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1680987405729 2023-04-08 20:56:53,094 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 47048196d3464a46a6c44457bdbde2ec, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1680987407905 2023-04-08 20:56:53,094 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting a44f51f2b80c4c898cfe5af94538c04c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1680987407905 2023-04-08 20:56:53,095 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting e8853eb9f0f64918a56f5b6f326d57a6, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1680987410061 2023-04-08 20:56:53,095 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting fdd43aa340584c03a1ecdc811a9c63fb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1680987410061 2023-04-08 20:56:53,095 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 419be3f2c9224c4bbf0c2ed1d9a5e7cf, keycount=250, bloomtype=ROW, size=38.7 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1680987410691 2023-04-08 20:56:53,095 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 58c388679c3b46d69c5df366bec67c7f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1680987410697 2023-04-08 20:56:53,112 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): eb4c5ed1339ded9a0ba1af8980e65ffc#B#compaction#323 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:56:53,115 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): eb4c5ed1339ded9a0ba1af8980e65ffc#A#compaction#322 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:56:53,123 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/B/602ad3539f16461fbd65ab93e087535c as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/602ad3539f16461fbd65ab93e087535c 2023-04-08 20:56:53,126 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in eb4c5ed1339ded9a0ba1af8980e65ffc/B of eb4c5ed1339ded9a0ba1af8980e65ffc into 602ad3539f16461fbd65ab93e087535c(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:56:53,126 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for eb4c5ed1339ded9a0ba1af8980e65ffc: 2023-04-08 20:56:53,126 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc., storeName=eb4c5ed1339ded9a0ba1af8980e65ffc/B, priority=9, startTime=1680987413089; duration=0sec 2023-04-08 20:56:53,127 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:56:53,127 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-08 20:56:53,129 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 80858 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-08 20:56:53,129 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): eb4c5ed1339ded9a0ba1af8980e65ffc/C is initiating minor compaction (all files) 2023-04-08 20:56:53,129 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of eb4c5ed1339ded9a0ba1af8980e65ffc/C in TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:53,129 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/8a302cb8e9104fa7a19d469e31e2316d, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/9a5f54890148437885355dc1c3c9b2ac, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/7fc987c4d213417bbb894481dcffe159, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/0a688d4ae27a4c0185ab6105c5893415, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/480ebefa348d459cb4b826f360134ae5, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/818a54bbaf484dca92c437bc61db2831, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/8b3b2580a6d4455182a60a9a5d6dc980] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp, totalSize=79.0 K 2023-04-08 20:56:53,130 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 8a302cb8e9104fa7a19d469e31e2316d, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680987402935 2023-04-08 20:56:53,130 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 9a5f54890148437885355dc1c3c9b2ac, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680987402951 2023-04-08 20:56:53,130 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 7fc987c4d213417bbb894481dcffe159, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680987405091 2023-04-08 20:56:53,130 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 0a688d4ae27a4c0185ab6105c5893415, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1680987405729 2023-04-08 20:56:53,131 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 480ebefa348d459cb4b826f360134ae5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1680987407905 2023-04-08 20:56:53,131 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 818a54bbaf484dca92c437bc61db2831, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1680987410061 2023-04-08 20:56:53,131 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 8b3b2580a6d4455182a60a9a5d6dc980, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1680987410697 2023-04-08 20:56:53,153 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:53,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:53,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:53,156 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): eb4c5ed1339ded9a0ba1af8980e65ffc#C#compaction#324 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:56:53,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:53,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:53,158 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:53,158 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:53,158 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:53,158 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:53,159 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:53,159 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:53,159 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:53,159 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:53,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:53,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:53,160 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:53,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:53,160 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:53,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:53,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:53,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:53,160 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:53,166 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:53,166 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:53,166 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:53,166 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:53,166 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:53,170 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:53,170 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:53,170 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:53,170 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:56:53,170 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing eb4c5ed1339ded9a0ba1af8980e65ffc 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-08 20:56:53,170 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=A 2023-04-08 20:56:53,170 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:53,170 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=B 2023-04-08 20:56:53,170 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:53,170 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=C 2023-04-08 20:56:53,170 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:53,181 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/C/494d814ab3ae4f6da2623e71a00f48bf as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/494d814ab3ae4f6da2623e71a00f48bf 2023-04-08 20:56:53,187 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in eb4c5ed1339ded9a0ba1af8980e65ffc/C of eb4c5ed1339ded9a0ba1af8980e65ffc into 494d814ab3ae4f6da2623e71a00f48bf(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:56:53,187 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for eb4c5ed1339ded9a0ba1af8980e65ffc: 2023-04-08 20:56:53,187 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc., storeName=eb4c5ed1339ded9a0ba1af8980e65ffc/C, priority=9, startTime=1680987413089; duration=0sec 2023-04-08 20:56:53,187 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:56:53,599 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408956fc13a237049748ee5267ad3745453 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408956fc13a237049748ee5267ad3745453 2023-04-08 20:56:53,935 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/A/494a1154e4964cc0bde42aa8c5a79ed4 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/494a1154e4964cc0bde42aa8c5a79ed4 2023-04-08 20:56:53,939 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in eb4c5ed1339ded9a0ba1af8980e65ffc/A of eb4c5ed1339ded9a0ba1af8980e65ffc into 494a1154e4964cc0bde42aa8c5a79ed4(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:56:53,939 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for eb4c5ed1339ded9a0ba1af8980e65ffc: 2023-04-08 20:56:53,939 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc., storeName=eb4c5ed1339ded9a0ba1af8980e65ffc/A, priority=9, startTime=1680987413089; duration=0sec 2023-04-08 20:56:53,939 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:56:54,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:56:54,002 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:56:54,003 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=134, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/A/c2f4ab59b27641edbc9bb858c6af5ad8 2023-04-08 20:56:54,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:54,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987474014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:54,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:54,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987474014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:54,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:54,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987474016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:54,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:54,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987474017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:54,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:54,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987474118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:54,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:54,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987474119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:54,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:54,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987474119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:54,124 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:54,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987474122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:54,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:54,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987474228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:54,230 DEBUG [Thread-2291] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4159 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc., hostname=jenkins-hbase20.apache.org,41091,1680987313601, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 20:56:54,322 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:54,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987474321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:54,323 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:54,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987474321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:54,323 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:54,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987474322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:54,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:54,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987474326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:54,411 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/B/c3bfa8c5b4c44c408738cf084b5e0770 2023-04-08 20:56:54,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:54,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987474624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:54,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:54,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987474624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:54,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:54,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987474625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:54,629 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:54,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987474627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:54,820 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/C/60f4ccf08ff8482fa9ec2938e130db47 2023-04-08 20:56:54,824 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/A/c2f4ab59b27641edbc9bb858c6af5ad8 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/c2f4ab59b27641edbc9bb858c6af5ad8 2023-04-08 20:56:54,827 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/c2f4ab59b27641edbc9bb858c6af5ad8, entries=150, sequenceid=134, filesize=25.2 K 2023-04-08 20:56:54,828 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/B/c3bfa8c5b4c44c408738cf084b5e0770 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/c3bfa8c5b4c44c408738cf084b5e0770 2023-04-08 20:56:54,832 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/c3bfa8c5b4c44c408738cf084b5e0770, entries=150, sequenceid=134, filesize=11.7 K 2023-04-08 20:56:54,833 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/C/60f4ccf08ff8482fa9ec2938e130db47 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/60f4ccf08ff8482fa9ec2938e130db47 2023-04-08 20:56:54,836 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/60f4ccf08ff8482fa9ec2938e130db47, entries=150, sequenceid=134, filesize=11.7 K 2023-04-08 20:56:54,837 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for eb4c5ed1339ded9a0ba1af8980e65ffc in 1667ms, sequenceid=134, compaction requested=false 2023-04-08 20:56:54,837 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for eb4c5ed1339ded9a0ba1af8980e65ffc: 2023-04-08 20:56:54,837 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:54,837 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:54,837 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:54,837 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:54,838 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:54,838 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:54,839 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:54,839 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:54,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:54,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:54,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:54,839 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:54,839 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:54,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:54,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:54,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:54,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:54,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:54,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:54,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@124abdea[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:54,840 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:54,840 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:54,841 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:54,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:54,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:54,841 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:54,841 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:54,841 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:54,842 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:54,842 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:54,842 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:54,842 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:54,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:54,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:54,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:54,843 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:54,843 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:54,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:54,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:54,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:54,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:54,846 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:54,846 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:54,846 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:54,846 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:54,846 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:54,864 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:54,864 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:54,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:54,864 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:54,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:54,865 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:54,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:54,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:54,866 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:54,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:54,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:54,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:54,873 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:54,873 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:54,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:54,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:54,873 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:54,873 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:54,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:54,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:54,873 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:54,874 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:54,874 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:54,874 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:54,874 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:54,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:54,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:54,874 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:54,874 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:54,874 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:54,974 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:54,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:54,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:54,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:54,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:54,977 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:54,977 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:54,977 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:54,977 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:54,977 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:54,977 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:54,978 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:54,978 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:54,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:54,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:54,978 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:54,978 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:54,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:54,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:54,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:54,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:54,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:54,979 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:54,979 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:54,979 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:54,979 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:54,979 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:54,979 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:54,979 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:54,979 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:54,979 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:56:54,979 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing eb4c5ed1339ded9a0ba1af8980e65ffc 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-08 20:56:54,980 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=A 2023-04-08 20:56:54,980 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:54,980 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=B 2023-04-08 20:56:54,980 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:54,980 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=C 2023-04-08 20:56:54,980 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:55,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:56:55,131 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:56:55,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:55,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987475146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:55,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:55,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987475147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:55,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:55,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987475148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:55,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:55,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987475149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:55,255 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:55,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987475252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:55,256 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:55,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987475252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:55,257 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:55,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987475254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:55,258 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:55,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987475255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:55,398 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304085d9b5db4236642ca952f5bbba3cf3e42 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304085d9b5db4236642ca952f5bbba3cf3e42 2023-04-08 20:56:55,461 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:55,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987475456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:55,461 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:55,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987475456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:55,461 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:55,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987475458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:55,461 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:55,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987475459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:55,764 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:55,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987475762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:55,764 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:55,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987475763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:55,764 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:55,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987475763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:55,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:55,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987475764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:55,803 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=150, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/A/12b1022a974a4040aa5e17b9387af524 2023-04-08 20:56:56,213 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/B/f082fef5cbe0456e8f0dc81028a99904 2023-04-08 20:56:56,271 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:56,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987476267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:56,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:56,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987476268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:56,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:56,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987476269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:56,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:56,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987476270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:56,621 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/C/b3d8515fd1f44bfd8a22131437d7aab5 2023-04-08 20:56:56,624 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/A/12b1022a974a4040aa5e17b9387af524 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/12b1022a974a4040aa5e17b9387af524 2023-04-08 20:56:56,628 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/12b1022a974a4040aa5e17b9387af524, entries=150, sequenceid=150, filesize=25.3 K 2023-04-08 20:56:56,629 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/B/f082fef5cbe0456e8f0dc81028a99904 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/f082fef5cbe0456e8f0dc81028a99904 2023-04-08 20:56:56,632 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/f082fef5cbe0456e8f0dc81028a99904, entries=150, sequenceid=150, filesize=11.8 K 2023-04-08 20:56:56,633 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/C/b3d8515fd1f44bfd8a22131437d7aab5 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/b3d8515fd1f44bfd8a22131437d7aab5 2023-04-08 20:56:56,636 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/b3d8515fd1f44bfd8a22131437d7aab5, entries=150, sequenceid=150, filesize=11.8 K 2023-04-08 20:56:56,637 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for eb4c5ed1339ded9a0ba1af8980e65ffc in 1658ms, sequenceid=150, compaction requested=true 2023-04-08 20:56:56,637 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for eb4c5ed1339ded9a0ba1af8980e65ffc: 2023-04-08 20:56:56,637 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:56,637 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:56,637 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:56,637 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:56,637 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:56,637 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:56,638 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:56,639 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:56,639 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:56,639 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:56,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:56,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:56,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:56,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:56,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:56,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:56,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:56,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:56,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:56,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23912139[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:56,641 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:56,641 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:56,641 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:56,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:56,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:56,641 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:56,641 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:56,641 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:56,641 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:56,641 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:56,642 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:56,642 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:56,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:56,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:56,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:56,642 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:56,642 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:56,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:56,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:56,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:56,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:56,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:56,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:56,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:56,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:56,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:56,644 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:56,644 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:56,644 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:56,644 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:56,644 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:56,645 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:56,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:56,645 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:56,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:56,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:56,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:56,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:56,645 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:56,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:56,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:56,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:56,645 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:56,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:56,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:56,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:56,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:56,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:56,646 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:56,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:56,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:56,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:56,648 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:56,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:56,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:56,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:56,648 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:56,648 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:56,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:56,648 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:56,648 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:56,648 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:56,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:56,648 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:56,648 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:56,648 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:56,648 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:56,648 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:56,648 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:56,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:56,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:56,748 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:56,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:56,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:56,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:56,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:56,753 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:56,753 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:56,753 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:56,753 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:56,753 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:56,753 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:56,754 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:56,754 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:56,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:56,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:56,754 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:56,754 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:56,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:56,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:56,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:56,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:56,755 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:56,755 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:56,755 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:56,755 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:56,755 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:56,755 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:56,755 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:56,755 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:56,755 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:56,755 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:56:56,755 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing eb4c5ed1339ded9a0ba1af8980e65ffc 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-08 20:56:56,755 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=A 2023-04-08 20:56:56,755 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:56,755 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=B 2023-04-08 20:56:56,756 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:56,756 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=C 2023-04-08 20:56:56,756 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:57,165 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408e9c7f68cc658458ab29b2c5fb9c2e9db to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408e9c7f68cc658458ab29b2c5fb9c2e9db 2023-04-08 20:56:57,170 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=168, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/A/5076a00470004d399fdc177761832b01 2023-04-08 20:56:57,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:56:57,279 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:56:57,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:57,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987477305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:57,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:57,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987477312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:57,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:57,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987477313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:57,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:57,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987477314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:57,417 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:57,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987477415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:57,422 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:57,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987477417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:57,425 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:57,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987477421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:57,425 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:57,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987477421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:57,584 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/B/1290c19e99684bdcbb3f4a593c4e6f55 2023-04-08 20:56:57,620 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:57,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987477618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:57,626 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:57,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987477624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:57,627 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:57,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987477626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:57,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:57,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987477627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:57,925 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:57,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987477922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:57,930 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:57,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987477927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:57,931 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:57,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987477928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:57,936 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:57,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987477933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:57,992 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/C/06e620299d8849668d851122b1ec26e3 2023-04-08 20:56:57,995 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/A/5076a00470004d399fdc177761832b01 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/5076a00470004d399fdc177761832b01 2023-04-08 20:56:57,998 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/5076a00470004d399fdc177761832b01, entries=150, sequenceid=168, filesize=25.3 K 2023-04-08 20:56:57,999 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/B/1290c19e99684bdcbb3f4a593c4e6f55 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/1290c19e99684bdcbb3f4a593c4e6f55 2023-04-08 20:56:58,002 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/1290c19e99684bdcbb3f4a593c4e6f55, entries=150, sequenceid=168, filesize=11.8 K 2023-04-08 20:56:58,002 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/C/06e620299d8849668d851122b1ec26e3 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/06e620299d8849668d851122b1ec26e3 2023-04-08 20:56:58,005 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/06e620299d8849668d851122b1ec26e3, entries=150, sequenceid=168, filesize=11.8 K 2023-04-08 20:56:58,006 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for eb4c5ed1339ded9a0ba1af8980e65ffc in 1251ms, sequenceid=168, compaction requested=true 2023-04-08 20:56:58,006 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for eb4c5ed1339ded9a0ba1af8980e65ffc: 2023-04-08 20:56:58,006 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:58,006 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:58,006 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:58,006 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:58,006 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:58,006 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:58,007 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:58,007 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:58,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:58,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:58,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:58,008 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:58,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:58,008 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:58,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:58,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:58,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:58,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:58,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:58,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cd318cd[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:58,009 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:58,009 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:58,009 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:58,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:58,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:58,009 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:58,010 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:58,010 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:58,010 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:58,010 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:58,010 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:58,010 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:58,010 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:58,010 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:58,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:58,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:58,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:58,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:58,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:58,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:58,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:58,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:58,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:58,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:58,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:58,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:58,012 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:58,012 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:58,012 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:58,012 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:58,012 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:58,014 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:58,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:58,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:58,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:58,015 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:58,015 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:58,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:58,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:58,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:58,023 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:58,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:58,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:58,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:58,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:58,024 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:58,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:58,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:58,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:58,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:58,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:58,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:58,033 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:58,033 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:58,033 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:58,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:58,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:58,033 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:58,033 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:58,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:58,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:58,033 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:58,033 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:58,033 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:58,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:58,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:58,033 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:58,033 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:58,033 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:58,033 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:58,133 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:58,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:58,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:58,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:58,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:58,136 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:58,136 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:58,136 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:58,136 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:58,136 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:58,136 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:58,137 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:58,137 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:58,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:58,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:58,137 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:58,137 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:58,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:58,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:58,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:58,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:58,138 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:58,138 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:58,138 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:58,138 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:58,138 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:58,138 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:58,138 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:58,138 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:58,138 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:56:58,139 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing eb4c5ed1339ded9a0ba1af8980e65ffc 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-08 20:56:58,139 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=A 2023-04-08 20:56:58,139 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:58,139 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=B 2023-04-08 20:56:58,139 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:58,139 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=C 2023-04-08 20:56:58,139 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:58,152 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408b63b5497689a4dafb3ad8a805ca4dd32 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408b63b5497689a4dafb3ad8a805ca4dd32 2023-04-08 20:56:58,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:56:58,256 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:56:58,326 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:58,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987478323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:58,432 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:58,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987478428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:58,432 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:58,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987478428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:58,434 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:58,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987478431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:58,435 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:58,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987478432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:58,446 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:58,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987478441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:58,556 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=182, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/A/75bee729dd374959b93376c75370214a 2023-04-08 20:56:58,636 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:58,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987478633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:58,942 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:58,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987478937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:58,966 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/B/310e728ef90c40fabdcae8c6da2168c6 2023-04-08 20:56:59,375 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/C/aea3be35b335475392ea1b4cada0b1c6 2023-04-08 20:56:59,378 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/A/75bee729dd374959b93376c75370214a as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/75bee729dd374959b93376c75370214a 2023-04-08 20:56:59,381 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/75bee729dd374959b93376c75370214a, entries=150, sequenceid=182, filesize=25.3 K 2023-04-08 20:56:59,382 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/B/310e728ef90c40fabdcae8c6da2168c6 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/310e728ef90c40fabdcae8c6da2168c6 2023-04-08 20:56:59,385 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/310e728ef90c40fabdcae8c6da2168c6, entries=150, sequenceid=182, filesize=11.8 K 2023-04-08 20:56:59,386 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/C/aea3be35b335475392ea1b4cada0b1c6 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/aea3be35b335475392ea1b4cada0b1c6 2023-04-08 20:56:59,389 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/aea3be35b335475392ea1b4cada0b1c6, entries=150, sequenceid=182, filesize=11.8 K 2023-04-08 20:56:59,389 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for eb4c5ed1339ded9a0ba1af8980e65ffc in 1251ms, sequenceid=182, compaction requested=true 2023-04-08 20:56:59,389 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for eb4c5ed1339ded9a0ba1af8980e65ffc: 2023-04-08 20:56:59,389 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:59,389 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:59,389 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:59,390 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,390 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:59,390 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:59,390 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,390 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:59,391 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,391 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:59,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:59,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5082416d[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:59,392 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:59,392 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,392 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,392 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:59,392 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:59,392 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:59,392 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,393 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:59,393 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,393 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:59,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:59,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:59,393 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:59,393 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:59,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:59,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:59,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:59,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:59,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:59,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:59,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:59,396 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:59,396 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:59,396 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:59,396 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:59,396 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:59,397 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,397 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:59,397 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:59,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:59,397 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:59,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:59,398 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:59,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:59,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,399 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,399 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:59,399 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:59,399 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,401 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,401 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,401 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,401 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:59,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:59,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:59,401 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:59,401 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:59,401 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:59,401 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:59,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:59,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:59,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:59,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:59,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:59,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:56:59,443 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing eb4c5ed1339ded9a0ba1af8980e65ffc 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-08 20:56:59,443 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=A 2023-04-08 20:56:59,443 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:59,443 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=B 2023-04-08 20:56:59,443 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:59,443 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=C 2023-04-08 20:56:59,443 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:56:59,472 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:59,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987479460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,472 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:59,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54766 deadline: 1680987479461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,472 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:59,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987479462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,473 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:59,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987479462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,473 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:59,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987479466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,501 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:59,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:59,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:59,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:59,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:59,505 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:59,505 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:59,505 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:59,505 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:59,505 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,505 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:59,506 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:59,506 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:59,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:59,506 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:59,506 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:59,506 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:59,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:59,507 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:59,507 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:59,507 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:59,507 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:59,507 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:59,507 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:59,507 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:59,507 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:59,507 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:59,507 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:56:59,507 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:56:59,507 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:59,507 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:59,508 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:59,508 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,508 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:59,508 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:59,508 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,508 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:59,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:59,509 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,509 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:59,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:59,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:59,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:59,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:59,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:59,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1760037a[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:59,510 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:59,510 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,510 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,510 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:59,510 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:59,510 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:59,511 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,511 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:59,511 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,511 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:59,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:59,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:59,511 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:59,511 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:59,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:59,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:59,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:59,513 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:59,513 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:59,513 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:59,513 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:59,513 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:59,514 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,514 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:59,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:59,514 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:59,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:59,514 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:59,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:59,517 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:59,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:59,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,518 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,518 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:59,518 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:59,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:59,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:59,518 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:59,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:59,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:59,518 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:59,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:59,518 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:59,518 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,518 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,518 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:59,518 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,518 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:59,518 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:59,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:59,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:59,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54798 deadline: 1680987479573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:59,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54766 deadline: 1680987479573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:59,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987479573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:59,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987479573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,578 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:59,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54772 deadline: 1680987479573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,618 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:59,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:59,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:59,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:59,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:59,621 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:59,621 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:59,621 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:59,621 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:59,622 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,622 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:59,622 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:59,622 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:59,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:59,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:59,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:59,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:59,623 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:59,623 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:59,623 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:59,623 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:59,623 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:59,623 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:59,623 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:59,623 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:59,623 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:59,624 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:56:59,624 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,624 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:59,624 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:56:59,624 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:59,624 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:59,624 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:59,624 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,624 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:59,624 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:59,625 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,625 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:59,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:59,625 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,625 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:59,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:59,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:59,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:59,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:59,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:59,626 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:59,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3dd27046[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:59,626 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,627 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,627 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:59,627 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:59,627 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,627 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:59,627 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:59,628 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,628 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:59,628 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:59,628 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:59,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:59,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:59,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:59,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:59,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:59,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:59,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:59,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:59,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:59,631 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:59,631 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:59,631 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:59,631 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:59,631 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:59,632 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,632 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:59,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:59,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,632 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:59,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:59,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:59,632 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:59,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:59,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:59,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:59,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,633 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:59,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:59,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,635 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:59,636 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:59,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:59,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:59,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:59,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:59,636 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,636 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:59,636 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:59,636 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,636 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,636 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,636 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,636 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:59,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:59,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:59,637 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:59,637 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:59,737 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:59,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:59,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:59,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:59,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:59,740 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:59,740 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:59,740 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:59,740 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:59,740 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,740 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:59,742 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,742 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:59,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:59,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:59,742 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,743 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:59,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:59,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:59,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:59,743 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:59,743 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:59,743 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:59,743 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:59,743 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:59,744 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:59,744 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:59,744 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:59,744 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:59,744 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:56:59,744 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:56:59,744 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:59,744 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:59,744 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:59,744 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,744 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:59,744 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:59,745 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,745 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:59,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:59,745 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,745 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:59,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:59,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:59,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:59,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:59,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:59,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79831241[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:59,746 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:59,746 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,747 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,747 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:59,747 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,747 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:59,747 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:59,747 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:59,748 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:59,748 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:59,748 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,748 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:59,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:59,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:59,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:59,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:59,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:59,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:59,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:59,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:59,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:59,749 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:59,749 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:59,749 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:59,749 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:59,749 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:59,750 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,750 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:59,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:59,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,750 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:59,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:59,751 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:59,752 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:59,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:59,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,753 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,753 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:59,753 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:59,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:59,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:59,754 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:59,754 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:59,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:59,754 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:59,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:59,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:59,754 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,754 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,754 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:59,754 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,754 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:59,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:59,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:59,754 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:59,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54766 deadline: 1680987479778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:59,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987479778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:59,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54772 deadline: 1680987479778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:59,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54798 deadline: 1680987479779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:56:59,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54780 deadline: 1680987479779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,854 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408b83efbed8af649f281ada019bb40ee81 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408b83efbed8af649f281ada019bb40ee81 2023-04-08 20:56:59,854 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:59,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:59,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:59,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:59,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:59,857 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:59,857 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:59,857 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:59,858 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:59,858 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,858 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:59,859 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,859 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:59,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:59,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:59,859 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,859 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:59,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:59,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:59,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:59,860 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:59,860 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:59,860 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:59,860 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:59,860 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:59,860 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:59,860 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:59,860 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:59,860 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:59,860 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:56:59,861 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:56:59,861 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:59,861 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:59,861 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:59,861 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,861 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:59,861 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:59,862 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,862 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:59,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:59,862 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:59,862 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:59,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:59,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:59,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:59,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:59,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b48bd35[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:59,865 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:59,865 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,865 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,866 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,866 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:59,866 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:59,866 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:59,866 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:59,866 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:59,866 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:59,866 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,866 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:59,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:59,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:59,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:59,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:59,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:59,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:59,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:59,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:59,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:59,868 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:59,868 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:59,868 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:59,868 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:59,868 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:59,869 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,869 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:59,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:59,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,869 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:59,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:59,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:59,870 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:59,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:59,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:59,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:59,870 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:59,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:59,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,872 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:59,872 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:59,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:59,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:59,872 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:56:59,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:59,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:59,872 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:59,873 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:59,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:59,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:59,873 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:59,873 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:59,873 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,873 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:59,873 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,873 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,873 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,973 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:56:59,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:56:59,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:56:59,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:56:59,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:56:59,976 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:56:59,976 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:59,976 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:56:59,976 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:56:59,976 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,976 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:59,977 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,977 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:59,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:59,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:59,977 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,977 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:56:59,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:59,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:56:59,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:59,978 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:56:59,978 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:56:59,978 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:56:59,978 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:56:59,978 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:56:59,978 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:56:59,978 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:56:59,978 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:56:59,978 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:59,978 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:56:59,978 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:56:59,978 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:56:59,978 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:56:59,978 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:56:59,978 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,978 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:56:59,978 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:56:59,979 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,979 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:59,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:59,980 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:59,980 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:56:59,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:59,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:59,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:59,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:56:59,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c2412f5[Count = 0] remaining members to acquire global barrier 2023-04-08 20:56:59,982 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:56:59,982 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,982 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,983 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,983 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:56:59,983 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:56:59,983 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:56:59,983 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:56:59,983 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:59,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:59,983 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:56:59,984 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:59,984 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:56:59,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:59,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:59,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:59,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:59,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:59,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:56:59,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:56:59,985 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:56:59,985 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:56:59,985 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:56:59,985 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:56:59,986 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:56:59,986 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,986 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:56:59,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:56:59,986 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:59,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:59,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:59,987 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:56:59,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:56:59,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:59,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:56:59,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:59,987 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:56:59,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:56:59,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,990 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,990 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:56:59,990 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:56:59,990 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:59,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:56:59,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:56:59,990 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:56:59,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:56:59,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:56:59,990 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:56:59,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:56:59,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:56:59,990 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:56:59,990 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,990 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:56:59,990 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:56:59,990 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:56:59,990 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,086 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:00,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54766 deadline: 1680987480085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,090 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:00,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:00,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:00,092 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:00,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54798 deadline: 1680987480087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,092 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:00,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987480087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:00,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:00,093 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:00,093 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:00,093 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:00,093 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:00,093 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,093 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:00,093 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54772 deadline: 1680987480087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,093 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:00,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54780 deadline: 1680987480087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,094 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,094 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:00,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:00,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:00,094 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,094 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:00,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:00,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,095 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:00,095 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:00,095 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:00,095 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:00,095 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:00,095 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:00,096 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:00,096 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:00,096 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:57:00,096 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:57:00,096 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:57:00,096 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:57:00,096 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:00,096 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:00,096 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,096 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:00,096 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:00,097 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,097 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:00,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:00,097 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:00,097 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:00,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:00,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:00,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:00,098 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:00,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2da0777[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:00,098 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,098 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,098 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:00,099 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:00,099 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,099 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:00,099 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:00,099 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,099 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:00,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:00,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:00,099 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:00,099 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:00,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:00,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:00,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:00,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:00,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:00,101 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:00,101 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:00,101 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:00,101 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:00,101 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:00,102 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,102 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:00,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:00,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,102 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:00,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:00,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:00,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:00,103 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:00,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,103 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:00,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,105 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,105 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,105 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:00,105 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:00,105 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:00,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:00,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:00,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:00,105 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:00,105 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:00,105 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,105 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:00,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:00,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:00,105 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:00,105 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,105 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,205 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:00,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:00,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:00,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:00,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:00,208 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:00,208 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:00,208 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:00,208 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:00,209 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,209 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,209 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,209 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:00,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:00,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:00,210 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,210 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:00,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:00,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,210 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:00,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:00,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:00,210 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:00,210 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:00,211 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:00,211 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:00,211 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:00,211 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:57:00,211 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:57:00,211 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:57:00,211 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:57:00,211 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:00,211 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:00,211 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,211 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:00,211 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:00,212 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,212 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:00,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:00,212 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,212 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:00,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:00,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:00,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:00,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:00,213 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:00,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63aa1c77[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:00,213 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,213 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,213 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,213 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:00,213 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:00,213 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:00,213 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:00,214 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:00,214 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,214 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:00,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:00,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:00,214 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:00,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:00,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:00,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:00,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:00,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:00,216 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:00,216 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:00,216 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:00,216 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:00,216 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:00,217 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:00,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:00,217 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,217 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:00,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:00,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:00,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:00,218 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:00,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,219 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:00,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,220 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,220 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,220 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,220 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,220 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:00,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:00,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:00,220 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:00,220 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,220 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:00,221 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:00,221 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:00,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:00,221 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:00,221 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:00,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:00,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:00,259 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=201, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/A/a33497b80abc46f18eba38d369e06af7 2023-04-08 20:57:00,321 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:00,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:00,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:00,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:00,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:00,324 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:00,324 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:00,324 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:00,324 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:00,324 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,324 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,325 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,325 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:00,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:00,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:00,325 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,325 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:00,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:00,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,326 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:00,326 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:00,326 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:00,326 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:00,326 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:00,326 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:00,326 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:00,326 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:00,326 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:57:00,326 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:57:00,326 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:57:00,326 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:57:00,326 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:00,326 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:00,326 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,326 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:00,326 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:00,327 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,327 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:00,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:00,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:00,327 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,328 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:00,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:00,328 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:00,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d75b422[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:00,328 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,329 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,329 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,329 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:00,329 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:00,329 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:00,329 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:00,330 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,330 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:00,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:00,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:00,330 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:00,330 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:00,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:00,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:00,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:00,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:00,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:00,332 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:00,332 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:00,332 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:00,332 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:00,332 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:00,333 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,333 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:00,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,333 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:00,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:00,333 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:00,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,334 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:00,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,335 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,335 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:00,335 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,335 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:00,335 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:00,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:00,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:00,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:00,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:00,335 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,336 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,336 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,336 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,336 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:00,336 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:00,336 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:00,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:00,436 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:00,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:00,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:00,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:00,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:00,439 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:00,439 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:00,439 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:00,439 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:00,439 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,439 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,440 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,440 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:00,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:00,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:00,440 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,440 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:00,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:00,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,441 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:00,441 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:00,441 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:00,441 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:00,441 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:00,441 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:00,441 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:00,441 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:00,441 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:57:00,441 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:57:00,441 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:57:00,441 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:57:00,441 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:00,441 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:00,441 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,441 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:00,441 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:00,442 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,442 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:00,442 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,442 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:00,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:00,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11caaec3[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:00,443 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:00,444 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,445 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,445 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:00,445 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:00,445 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:00,445 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,445 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:00,446 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,446 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:00,446 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:00,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:00,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:00,446 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:00,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:00,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:00,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:00,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:00,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:00,447 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:00,447 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:00,447 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:00,447 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:00,448 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:00,448 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:00,449 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,449 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:00,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:00,449 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:00,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:00,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:00,450 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:00,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,451 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,451 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:00,451 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:00,451 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:00,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:00,451 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:00,451 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:00,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:00,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:00,451 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,452 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,452 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,452 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:00,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:00,452 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:00,452 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:00,552 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:00,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:00,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:00,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:00,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:00,555 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:00,555 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:00,555 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:00,555 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:00,555 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,555 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,556 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:00,556 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:00,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:00,556 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,556 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:00,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:00,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,557 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:00,557 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:00,557 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:00,557 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:00,557 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:00,557 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:00,557 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:00,557 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:00,557 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:57:00,557 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:57:00,557 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:57:00,557 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:57:00,557 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:00,557 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:00,557 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,557 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:00,557 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:00,558 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,558 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:00,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:00,559 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:00,559 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:00,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:00,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:00,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:00,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4379e232[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:00,560 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:00,561 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,561 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,561 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:00,561 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,561 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:00,561 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:00,561 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:00,562 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,562 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:00,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:00,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:00,562 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:00,562 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:00,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:00,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:00,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:00,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:00,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:00,564 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:00,564 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:00,564 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:00,564 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:00,564 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:00,564 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,565 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,565 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:00,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:00,565 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:00,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,566 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:00,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,568 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,568 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:00,568 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,568 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:00,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:00,568 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:00,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:00,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:00,568 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,568 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,568 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:00,568 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,568 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,568 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:00,568 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:00,595 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:00,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54766 deadline: 1680987480591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,599 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:00,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54780 deadline: 1680987480594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,599 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:00,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987480594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,602 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:00,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54798 deadline: 1680987480597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,602 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:00,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54772 deadline: 1680987480598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,668 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=201 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/B/f529d112ceb34b4daf65b21dfbfa8eb2 2023-04-08 20:57:00,668 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:00,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:00,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:00,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:00,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:00,671 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:00,671 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:00,671 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:00,671 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:00,671 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,672 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,672 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,672 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:00,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:00,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:00,672 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,672 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:00,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:00,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,678 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:00,678 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:00,678 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:00,678 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:00,678 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:00,678 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:00,679 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:00,679 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:00,679 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:57:00,679 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:57:00,679 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:57:00,679 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:57:00,679 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:00,679 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:00,679 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,679 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:00,679 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:00,680 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,680 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:00,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:00,680 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,680 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:00,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:00,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:00,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:00,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:00,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7083178e[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:00,682 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:00,682 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,683 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,683 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:00,683 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:00,683 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:00,683 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,683 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:00,684 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:00,684 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:00,684 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:00,686 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:00,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:00,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:00,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:00,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:00,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:00,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:00,688 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:00,688 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:00,688 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:00,688 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:00,688 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:00,697 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,697 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,697 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:00,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:00,698 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:00,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,699 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:00,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,701 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:00,701 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:00,701 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:00,701 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:00,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:00,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:00,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:00,701 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,701 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,701 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,701 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,701 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:00,701 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:00,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:00,701 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,701 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:00,801 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:00,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:00,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:00,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:00,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:00,804 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:00,804 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:00,804 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:00,804 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:00,805 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,805 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,805 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,805 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:00,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:00,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:00,805 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,805 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:00,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:00,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,806 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:00,806 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:00,806 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:00,806 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:00,806 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:00,806 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:00,806 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:00,806 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:00,806 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:57:00,806 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:57:00,807 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:57:00,807 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:57:00,807 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:00,807 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:00,807 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,807 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:00,807 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:00,807 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,808 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:00,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:00,808 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,808 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:00,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:00,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:00,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:00,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:00,809 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:00,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40676918[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:00,809 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,809 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,810 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:00,810 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:00,810 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:00,810 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,810 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:00,811 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,811 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:00,811 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:00,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:00,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:00,811 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:00,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:00,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:00,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:00,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:00,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:00,813 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:00,813 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:00,813 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:00,813 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:00,813 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:00,814 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,814 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:00,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:00,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,814 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:00,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:00,814 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:00,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:00,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:00,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,815 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:00,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,817 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,817 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,817 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:00,817 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:00,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:00,817 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:00,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:00,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:00,817 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:00,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:00,817 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,817 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,817 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,817 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,817 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:00,817 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:00,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:00,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:00,917 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:00,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:00,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:00,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:00,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:00,920 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:00,920 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:00,920 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:00,920 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:00,920 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,921 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,923 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,923 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:00,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:00,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:00,924 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,924 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:00,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:00,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,924 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:00,924 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:00,924 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:00,924 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:00,925 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:00,925 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:00,925 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:00,925 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:00,925 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:57:00,925 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:57:00,925 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:57:00,925 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:57:00,925 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:00,925 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:00,925 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,925 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:00,925 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:00,926 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,926 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:00,926 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,926 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:00,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:00,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:00,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41f6b61[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:00,927 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:00,927 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,928 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,928 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,928 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:00,928 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:00,928 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:00,928 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:00,929 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,929 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:00,929 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:00,929 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:00,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:00,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:00,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:00,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:00,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:00,931 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:00,931 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:00,931 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:00,931 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:00,931 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:00,932 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,932 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,932 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:00,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:00,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:00,932 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:00,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:00,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:00,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:00,933 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:00,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:00,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,934 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,934 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,934 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:00,934 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:00,935 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:00,934 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:00,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:00,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:00,935 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:00,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:00,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:00,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:00,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:00,935 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,935 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:00,935 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:00,935 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:00,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:00,935 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:01,037 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:01,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:01,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:01,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:01,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:01,040 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:01,040 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:01,040 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:01,040 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:01,040 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:01,040 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:01,041 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:01,041 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:01,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:01,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:01,041 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:01,041 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:01,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:01,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:01,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:01,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:01,041 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:01,042 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:01,042 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:01,042 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:01,042 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:01,042 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:01,042 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:01,042 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:57:01,042 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:57:01,042 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:57:01,042 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:57:01,042 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:01,042 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:01,042 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:01,042 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:01,042 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:01,043 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:01,043 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:01,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:01,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:01,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:01,043 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:01,043 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:01,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:01,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:01,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:01,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:01,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:01,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:01,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@87b61a2[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:01,045 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:01,045 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:01,045 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:01,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:01,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:01,045 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:01,045 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:01,045 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:01,045 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:01,045 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:01,046 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:01,046 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:01,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:01,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:01,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:01,046 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:01,046 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:01,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:01,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:01,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:01,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:01,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:01,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:01,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:01,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:01,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:01,048 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:01,048 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:01,048 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:01,048 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:01,048 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:01,048 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:01,048 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:01,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:01,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:01,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:01,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:01,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:01,048 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:01,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:01,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:01,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:01,049 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:01,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:01,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:01,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:01,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:01,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:01,050 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:01,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:01,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:01,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:01,051 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:01,051 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:01,051 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:01,051 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:01,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:01,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:01,051 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:01,051 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:01,051 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:01,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:01,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:01,051 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:01,051 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:01,051 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:01,051 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:01,051 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:01,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:01,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:01,084 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=201 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/C/0a62d1c91ad24d30841531ea3d84b4ec 2023-04-08 20:57:01,088 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/A/a33497b80abc46f18eba38d369e06af7 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/a33497b80abc46f18eba38d369e06af7 2023-04-08 20:57:01,090 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/a33497b80abc46f18eba38d369e06af7, entries=200, sequenceid=201, filesize=32.1 K 2023-04-08 20:57:01,091 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/B/f529d112ceb34b4daf65b21dfbfa8eb2 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/f529d112ceb34b4daf65b21dfbfa8eb2 2023-04-08 20:57:01,095 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/f529d112ceb34b4daf65b21dfbfa8eb2, entries=150, sequenceid=201, filesize=11.8 K 2023-04-08 20:57:01,095 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/C/0a62d1c91ad24d30841531ea3d84b4ec as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/0a62d1c91ad24d30841531ea3d84b4ec 2023-04-08 20:57:01,098 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/0a62d1c91ad24d30841531ea3d84b4ec, entries=150, sequenceid=201, filesize=11.8 K 2023-04-08 20:57:01,099 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for eb4c5ed1339ded9a0ba1af8980e65ffc in 1656ms, sequenceid=201, compaction requested=true 2023-04-08 20:57:01,099 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for eb4c5ed1339ded9a0ba1af8980e65ffc: 2023-04-08 20:57:01,099 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:57:01,099 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-08 20:57:01,099 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-08 20:57:01,099 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:57:01,099 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:57:01,100 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 162406 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-08 20:57:01,100 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72256 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-08 20:57:01,100 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): eb4c5ed1339ded9a0ba1af8980e65ffc/A is initiating minor compaction (all files) 2023-04-08 20:57:01,100 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): eb4c5ed1339ded9a0ba1af8980e65ffc/B is initiating minor compaction (all files) 2023-04-08 20:57:01,100 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of eb4c5ed1339ded9a0ba1af8980e65ffc/A in TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:57:01,100 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of eb4c5ed1339ded9a0ba1af8980e65ffc/B in TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:57:01,100 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/602ad3539f16461fbd65ab93e087535c, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/c3bfa8c5b4c44c408738cf084b5e0770, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/f082fef5cbe0456e8f0dc81028a99904, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/1290c19e99684bdcbb3f4a593c4e6f55, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/310e728ef90c40fabdcae8c6da2168c6, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/f529d112ceb34b4daf65b21dfbfa8eb2] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp, totalSize=70.6 K 2023-04-08 20:57:01,100 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/494a1154e4964cc0bde42aa8c5a79ed4, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/c2f4ab59b27641edbc9bb858c6af5ad8, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/12b1022a974a4040aa5e17b9387af524, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/5076a00470004d399fdc177761832b01, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/75bee729dd374959b93376c75370214a, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/a33497b80abc46f18eba38d369e06af7] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp, totalSize=158.6 K 2023-04-08 20:57:01,101 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 602ad3539f16461fbd65ab93e087535c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1680987410697 2023-04-08 20:57:01,101 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 494a1154e4964cc0bde42aa8c5a79ed4, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1680987410697 2023-04-08 20:57:01,101 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting c3bfa8c5b4c44c408738cf084b5e0770, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1680987411850 2023-04-08 20:57:01,101 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting c2f4ab59b27641edbc9bb858c6af5ad8, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1680987411850 2023-04-08 20:57:01,101 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting f082fef5cbe0456e8f0dc81028a99904, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1680987414013 2023-04-08 20:57:01,101 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 12b1022a974a4040aa5e17b9387af524, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1680987414013 2023-04-08 20:57:01,101 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 1290c19e99684bdcbb3f4a593c4e6f55, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1680987415146 2023-04-08 20:57:01,101 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 5076a00470004d399fdc177761832b01, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1680987415146 2023-04-08 20:57:01,102 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 310e728ef90c40fabdcae8c6da2168c6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1680987417289 2023-04-08 20:57:01,102 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 75bee729dd374959b93376c75370214a, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1680987417289 2023-04-08 20:57:01,102 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting f529d112ceb34b4daf65b21dfbfa8eb2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1680987418311 2023-04-08 20:57:01,102 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting a33497b80abc46f18eba38d369e06af7, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1680987418300 2023-04-08 20:57:01,115 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): eb4c5ed1339ded9a0ba1af8980e65ffc#B#compaction#340 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:57:01,121 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): eb4c5ed1339ded9a0ba1af8980e65ffc#A#compaction#341 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:57:01,125 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/B/1fedfa15cabf4aa881355ec6457b18e7 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/1fedfa15cabf4aa881355ec6457b18e7 2023-04-08 20:57:01,128 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in eb4c5ed1339ded9a0ba1af8980e65ffc/B of eb4c5ed1339ded9a0ba1af8980e65ffc into 1fedfa15cabf4aa881355ec6457b18e7(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:57:01,129 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for eb4c5ed1339ded9a0ba1af8980e65ffc: 2023-04-08 20:57:01,129 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc., storeName=eb4c5ed1339ded9a0ba1af8980e65ffc/B, priority=10, startTime=1680987421099; duration=0sec 2023-04-08 20:57:01,129 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:57:01,129 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-08 20:57:01,130 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72256 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-08 20:57:01,130 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): eb4c5ed1339ded9a0ba1af8980e65ffc/C is initiating minor compaction (all files) 2023-04-08 20:57:01,130 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of eb4c5ed1339ded9a0ba1af8980e65ffc/C in TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:57:01,130 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/494d814ab3ae4f6da2623e71a00f48bf, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/60f4ccf08ff8482fa9ec2938e130db47, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/b3d8515fd1f44bfd8a22131437d7aab5, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/06e620299d8849668d851122b1ec26e3, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/aea3be35b335475392ea1b4cada0b1c6, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/0a62d1c91ad24d30841531ea3d84b4ec] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp, totalSize=70.6 K 2023-04-08 20:57:01,131 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 494d814ab3ae4f6da2623e71a00f48bf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1680987410697 2023-04-08 20:57:01,131 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 60f4ccf08ff8482fa9ec2938e130db47, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1680987411850 2023-04-08 20:57:01,131 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting b3d8515fd1f44bfd8a22131437d7aab5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1680987414013 2023-04-08 20:57:01,131 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 06e620299d8849668d851122b1ec26e3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1680987415146 2023-04-08 20:57:01,131 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting aea3be35b335475392ea1b4cada0b1c6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1680987417289 2023-04-08 20:57:01,132 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 0a62d1c91ad24d30841531ea3d84b4ec, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1680987418311 2023-04-08 20:57:01,143 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): eb4c5ed1339ded9a0ba1af8980e65ffc#C#compaction#342 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:57:01,152 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:01,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:01,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:01,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:01,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:01,158 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:01,158 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:01,158 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:01,158 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:01,159 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:01,159 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:01,159 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:01,159 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:01,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:01,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:01,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:01,160 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:01,160 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:01,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:01,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:01,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:01,161 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:01,161 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:01,161 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:01,161 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:01,161 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:01,161 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:01,165 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:01,165 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:57:01,165 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:01,165 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. started... 2023-04-08 20:57:01,165 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing eb4c5ed1339ded9a0ba1af8980e65ffc 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-08 20:57:01,165 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=A 2023-04-08 20:57:01,165 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:01,165 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=B 2023-04-08 20:57:01,165 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:01,165 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=C 2023-04-08 20:57:01,165 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:01,564 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/C/4431b1998c45423f95bcb89625347372 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/4431b1998c45423f95bcb89625347372 2023-04-08 20:57:01,567 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in eb4c5ed1339ded9a0ba1af8980e65ffc/C of eb4c5ed1339ded9a0ba1af8980e65ffc into 4431b1998c45423f95bcb89625347372(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:57:01,567 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for eb4c5ed1339ded9a0ba1af8980e65ffc: 2023-04-08 20:57:01,567 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc., storeName=eb4c5ed1339ded9a0ba1af8980e65ffc/C, priority=10, startTime=1680987421099; duration=0sec 2023-04-08 20:57:01,568 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:57:01,578 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304086fa79621714c4fbb925e64ba63b61028 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304086fa79621714c4fbb925e64ba63b61028 2023-04-08 20:57:01,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:57:01,598 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. as already flushing 2023-04-08 20:57:01,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:01,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54766 deadline: 1680987481616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:01,625 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:01,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54772 deadline: 1680987481619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:01,625 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:01,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54780 deadline: 1680987481623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:01,631 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:01,631 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:01,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54798 deadline: 1680987481624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:01,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987481625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:01,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:01,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54766 deadline: 1680987481725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:01,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:01,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54772 deadline: 1680987481726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:01,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:01,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54780 deadline: 1680987481726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:01,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:01,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987481732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:01,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:01,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54798 deadline: 1680987481732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:01,934 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:01,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54766 deadline: 1680987481930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:01,935 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:01,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54772 deadline: 1680987481930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:01,935 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:01,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54780 deadline: 1680987481931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:01,941 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:01,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987481936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:01,941 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:01,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54798 deadline: 1680987481936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:01,988 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=215, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/A/1b7ecf1a0da740f992a9ad06c55a705d 2023-04-08 20:57:02,240 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:02,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54772 deadline: 1680987482236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:02,240 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:02,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54766 deadline: 1680987482236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:02,240 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:02,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54780 deadline: 1680987482238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:02,247 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:02,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54798 deadline: 1680987482242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:02,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:02,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987482244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:02,362 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/A/d3e2dca96ace419cb1fbb1550d66a188 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/d3e2dca96ace419cb1fbb1550d66a188 2023-04-08 20:57:02,366 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in eb4c5ed1339ded9a0ba1af8980e65ffc/A of eb4c5ed1339ded9a0ba1af8980e65ffc into d3e2dca96ace419cb1fbb1550d66a188(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-08 20:57:02,366 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for eb4c5ed1339ded9a0ba1af8980e65ffc: 2023-04-08 20:57:02,366 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc., storeName=eb4c5ed1339ded9a0ba1af8980e65ffc/A, priority=10, startTime=1680987421099; duration=1sec 2023-04-08 20:57:02,366 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:57:02,398 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/B/240c38604580442bbb8d1fc9a28614be 2023-04-08 20:57:02,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:02,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54766 deadline: 1680987482744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:02,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:02,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54772 deadline: 1680987482745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:02,749 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:02,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54780 deadline: 1680987482745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:02,750 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:02,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54798 deadline: 1680987482750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:02,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:02,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54784 deadline: 1680987482752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:02,806 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/C/79ab0b2cc7114473b3ffbffdde33cc3c 2023-04-08 20:57:02,810 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/A/1b7ecf1a0da740f992a9ad06c55a705d as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/1b7ecf1a0da740f992a9ad06c55a705d 2023-04-08 20:57:02,813 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/1b7ecf1a0da740f992a9ad06c55a705d, entries=150, sequenceid=215, filesize=25.3 K 2023-04-08 20:57:02,814 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/B/240c38604580442bbb8d1fc9a28614be as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/240c38604580442bbb8d1fc9a28614be 2023-04-08 20:57:02,820 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/240c38604580442bbb8d1fc9a28614be, entries=150, sequenceid=215, filesize=11.8 K 2023-04-08 20:57:02,820 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/C/79ab0b2cc7114473b3ffbffdde33cc3c as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/79ab0b2cc7114473b3ffbffdde33cc3c 2023-04-08 20:57:02,824 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/79ab0b2cc7114473b3ffbffdde33cc3c, entries=150, sequenceid=215, filesize=11.8 K 2023-04-08 20:57:02,825 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for eb4c5ed1339ded9a0ba1af8980e65ffc in 1660ms, sequenceid=215, compaction requested=false 2023-04-08 20:57:02,825 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for eb4c5ed1339ded9a0ba1af8980e65ffc: 2023-04-08 20:57:02,825 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:57:02,825 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:02,825 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:02,825 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:02,826 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:02,826 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:02,827 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:02,827 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:02,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:02,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:02,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:02,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:02,827 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:02,827 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:02,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:02,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:02,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:02,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:02,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:02,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@602c2844[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:02,829 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:02,829 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:02,830 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:02,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:02,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:02,830 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:02,830 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:02,830 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:02,832 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:02,832 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:02,832 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:02,832 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:02,832 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:02,833 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:02,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:02,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:02,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:02,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:02,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:02,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:02,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:02,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:02,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:02,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:02,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:02,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:02,836 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:02,836 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:02,836 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:02,836 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:02,836 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:02,837 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:02,837 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:02,837 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:02,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:02,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:02,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:02,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:02,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:02,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:02,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:02,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:02,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:02,839 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:02,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:02,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:02,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:02,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:02,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:02,840 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:02,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:02,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:02,848 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:02,848 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:02,848 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:02,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:02,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:02,848 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:02,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:02,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:02,848 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:02,848 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:02,849 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:02,849 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:02,849 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:02,849 DEBUG [Thread-2299] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:02,849 DEBUG [Thread-2299] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:02,849 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:02,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:02,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:02,896 DEBUG [Thread-2300] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x525c59d1 to 127.0.0.1:52440 2023-04-08 20:57:02,896 DEBUG [Thread-2300] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:57:02,897 DEBUG [Thread-2304] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1747f9fc to 127.0.0.1:52440 2023-04-08 20:57:02,897 DEBUG [Thread-2304] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:57:02,897 DEBUG [Thread-2308] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x55dc208b to 127.0.0.1:52440 2023-04-08 20:57:02,897 DEBUG [Thread-2308] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:57:02,901 DEBUG [Thread-2302] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x078ba718 to 127.0.0.1:52440 2023-04-08 20:57:02,901 DEBUG [Thread-2302] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:57:02,902 DEBUG [Thread-2306] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3098b23b to 127.0.0.1:52440 2023-04-08 20:57:02,902 DEBUG [Thread-2306] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:57:02,949 DEBUG [Thread-2299] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:02,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:03,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:57:03,755 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing eb4c5ed1339ded9a0ba1af8980e65ffc 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-08 20:57:03,755 DEBUG [Thread-2297] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6d951686 to 127.0.0.1:52440 2023-04-08 20:57:03,755 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=A 2023-04-08 20:57:03,756 DEBUG [Thread-2293] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4e90d228 to 127.0.0.1:52440 2023-04-08 20:57:03,755 DEBUG [Thread-2297] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:57:03,756 DEBUG [Thread-2293] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:57:03,756 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:57:03,757 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=B 2023-04-08 20:57:03,757 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:57:03,757 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=C 2023-04-08 20:57:03,757 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:57:03,758 DEBUG [Thread-2295] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1aec6811 to 127.0.0.1:52440 2023-04-08 20:57:03,758 DEBUG [Thread-2295] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:57:03,759 DEBUG [Thread-2289] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x48681e8f to 127.0.0.1:52440 2023-04-08 20:57:03,759 DEBUG [Thread-2289] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:57:03,763 DEBUG [Thread-2291] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x342c272f to 127.0.0.1:52440 2023-04-08 20:57:03,763 DEBUG [Thread-2291] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:57:03,763 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-08 20:57:03,763 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(394): wrote 41 2023-04-08 20:57:03,763 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(394): wrote 37 2023-04-08 20:57:03,763 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(394): wrote 37 2023-04-08 20:57:03,763 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(394): wrote 38 2023-04-08 20:57:03,763 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(394): wrote 36 2023-04-08 20:57:03,763 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-08 20:57:03,763 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-08 20:57:03,763 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(402): scanned 2292 2023-04-08 20:57:03,763 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(403): verified 6871 rows 2023-04-08 20:57:03,763 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(402): scanned 2296 2023-04-08 20:57:03,763 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(403): verified 6887 rows 2023-04-08 20:57:03,763 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(402): scanned 2279 2023-04-08 20:57:03,763 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(403): verified 6836 rows 2023-04-08 20:57:03,763 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(402): scanned 2280 2023-04-08 20:57:03,764 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(403): verified 6835 rows 2023-04-08 20:57:03,764 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(402): scanned 2282 2023-04-08 20:57:03,764 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(403): verified 6845 rows 2023-04-08 20:57:03,764 INFO [Listener at localhost.localdomain/41823] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-08 20:57:03,764 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f29a773 to 127.0.0.1:52440 2023-04-08 20:57:03,764 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:57:03,766 INFO [Listener at localhost.localdomain/41823] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-08 20:57:03,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-08 20:57:03,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-08 20:57:03,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-08 20:57:03,770 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680987423770"}]},"ts":"1680987423770"} 2023-04-08 20:57:03,771 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-08 20:57:03,773 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-08 20:57:03,773 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040885f5e005ea864e7baeb69b9171543fd7 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040885f5e005ea864e7baeb69b9171543fd7 2023-04-08 20:57:03,773 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=eb4c5ed1339ded9a0ba1af8980e65ffc, UNASSIGN}] 2023-04-08 20:57:03,775 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=eb4c5ed1339ded9a0ba1af8980e65ffc, UNASSIGN 2023-04-08 20:57:03,775 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=eb4c5ed1339ded9a0ba1af8980e65ffc, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:03,775 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987423775"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680987423775"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680987423775"}]},"ts":"1680987423775"} 2023-04-08 20:57:03,776 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601}] 2023-04-08 20:57:03,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-08 20:57:03,928 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:57:03,928 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing eb4c5ed1339ded9a0ba1af8980e65ffc, disabling compactions & flushes 2023-04-08 20:57:03,928 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:57:04,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-08 20:57:04,178 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=238, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/A/f4b248bd8c074d69a31e8bb04de65570 2023-04-08 20:57:04,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-08 20:57:04,594 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/B/b6985a1222d94370b70c90e5fb95f3a9 2023-04-08 20:57:04,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-08 20:57:05,013 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/C/aabd594fcf6a4776898cb15eb4009ed5 2023-04-08 20:57:05,021 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/A/f4b248bd8c074d69a31e8bb04de65570 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/f4b248bd8c074d69a31e8bb04de65570 2023-04-08 20:57:05,028 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/f4b248bd8c074d69a31e8bb04de65570, entries=150, sequenceid=238, filesize=25.3 K 2023-04-08 20:57:05,030 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/B/b6985a1222d94370b70c90e5fb95f3a9 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/b6985a1222d94370b70c90e5fb95f3a9 2023-04-08 20:57:05,036 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/b6985a1222d94370b70c90e5fb95f3a9, entries=150, sequenceid=238, filesize=11.8 K 2023-04-08 20:57:05,037 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/C/aabd594fcf6a4776898cb15eb4009ed5 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/aabd594fcf6a4776898cb15eb4009ed5 2023-04-08 20:57:05,045 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/aabd594fcf6a4776898cb15eb4009ed5, entries=150, sequenceid=238, filesize=11.8 K 2023-04-08 20:57:05,046 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=20.13 KB/20610 for eb4c5ed1339ded9a0ba1af8980e65ffc in 1291ms, sequenceid=238, compaction requested=true 2023-04-08 20:57:05,046 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for eb4c5ed1339ded9a0ba1af8980e65ffc: 2023-04-08 20:57:05,046 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:57:05,046 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:57:05,046 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:57:05,046 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. because compaction request was cancelled 2023-04-08 20:57:05,047 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. after waiting 0 ms 2023-04-08 20:57:05,047 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:57:05,046 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. because compaction request was cancelled 2023-04-08 20:57:05,046 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:57:05,047 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing eb4c5ed1339ded9a0ba1af8980e65ffc 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-08 20:57:05,047 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. because compaction request was cancelled 2023-04-08 20:57:05,047 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:57:05,047 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=A 2023-04-08 20:57:05,048 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:05,048 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=B 2023-04-08 20:57:05,048 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:05,048 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK eb4c5ed1339ded9a0ba1af8980e65ffc, store=C 2023-04-08 20:57:05,048 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:05,063 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304081ce2da41d8734a3d8e4e0522c3a33ecf to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304081ce2da41d8734a3d8e4e0522c3a33ecf 2023-04-08 20:57:05,470 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=244, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/A/ee34c4af59f440b7b114487b276d4657 2023-04-08 20:57:05,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-08 20:57:05,886 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/B/40b02c4c54674aa5a4368e0db8142d94 2023-04-08 20:57:06,305 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/C/48da29b3f20742e48ccd187943f3dc6d 2023-04-08 20:57:06,315 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/A/ee34c4af59f440b7b114487b276d4657 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/ee34c4af59f440b7b114487b276d4657 2023-04-08 20:57:06,319 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/ee34c4af59f440b7b114487b276d4657, entries=150, sequenceid=244, filesize=25.3 K 2023-04-08 20:57:06,319 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/B/40b02c4c54674aa5a4368e0db8142d94 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/40b02c4c54674aa5a4368e0db8142d94 2023-04-08 20:57:06,323 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/40b02c4c54674aa5a4368e0db8142d94, entries=150, sequenceid=244, filesize=11.8 K 2023-04-08 20:57:06,324 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/.tmp/C/48da29b3f20742e48ccd187943f3dc6d as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/48da29b3f20742e48ccd187943f3dc6d 2023-04-08 20:57:06,328 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/48da29b3f20742e48ccd187943f3dc6d, entries=150, sequenceid=244, filesize=11.8 K 2023-04-08 20:57:06,329 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for eb4c5ed1339ded9a0ba1af8980e65ffc in 1282ms, sequenceid=244, compaction requested=true 2023-04-08 20:57:06,345 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/639ce6bba81d4d2d8ca45d8c6f41fd74, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/4a3dca94131c4a1d8dcbd042d9fdebe7, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/0989381648ff4abfabb2800bdfc467d5, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/f1618df8c11b4e0bad0800a796be3418, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/47048196d3464a46a6c44457bdbde2ec, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/e8853eb9f0f64918a56f5b6f326d57a6, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/419be3f2c9224c4bbf0c2ed1d9a5e7cf, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/494a1154e4964cc0bde42aa8c5a79ed4, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/c2f4ab59b27641edbc9bb858c6af5ad8, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/12b1022a974a4040aa5e17b9387af524, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/5076a00470004d399fdc177761832b01, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/75bee729dd374959b93376c75370214a, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/a33497b80abc46f18eba38d369e06af7] to archive 2023-04-08 20:57:06,346 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 20:57:06,347 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/639ce6bba81d4d2d8ca45d8c6f41fd74 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/639ce6bba81d4d2d8ca45d8c6f41fd74 2023-04-08 20:57:06,349 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/4a3dca94131c4a1d8dcbd042d9fdebe7 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/4a3dca94131c4a1d8dcbd042d9fdebe7 2023-04-08 20:57:06,350 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/0989381648ff4abfabb2800bdfc467d5 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/0989381648ff4abfabb2800bdfc467d5 2023-04-08 20:57:06,351 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/f1618df8c11b4e0bad0800a796be3418 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/f1618df8c11b4e0bad0800a796be3418 2023-04-08 20:57:06,352 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/47048196d3464a46a6c44457bdbde2ec to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/47048196d3464a46a6c44457bdbde2ec 2023-04-08 20:57:06,354 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/e8853eb9f0f64918a56f5b6f326d57a6 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/e8853eb9f0f64918a56f5b6f326d57a6 2023-04-08 20:57:06,355 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/419be3f2c9224c4bbf0c2ed1d9a5e7cf to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/419be3f2c9224c4bbf0c2ed1d9a5e7cf 2023-04-08 20:57:06,356 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/494a1154e4964cc0bde42aa8c5a79ed4 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/494a1154e4964cc0bde42aa8c5a79ed4 2023-04-08 20:57:06,357 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/c2f4ab59b27641edbc9bb858c6af5ad8 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/c2f4ab59b27641edbc9bb858c6af5ad8 2023-04-08 20:57:06,358 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/12b1022a974a4040aa5e17b9387af524 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/12b1022a974a4040aa5e17b9387af524 2023-04-08 20:57:06,360 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/5076a00470004d399fdc177761832b01 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/5076a00470004d399fdc177761832b01 2023-04-08 20:57:06,361 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/75bee729dd374959b93376c75370214a to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/75bee729dd374959b93376c75370214a 2023-04-08 20:57:06,362 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/a33497b80abc46f18eba38d369e06af7 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/a33497b80abc46f18eba38d369e06af7 2023-04-08 20:57:06,378 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/2563655cd3564c919f057633491c42c4, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/44a2955b55dc48a08a27f3dfe2f32859, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/965b8359c91e4630ba7366f7c2032b2f, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/bbf254924bdf4b49a8db43108a5eee2f, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/a44f51f2b80c4c898cfe5af94538c04c, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/fdd43aa340584c03a1ecdc811a9c63fb, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/602ad3539f16461fbd65ab93e087535c, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/58c388679c3b46d69c5df366bec67c7f, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/c3bfa8c5b4c44c408738cf084b5e0770, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/f082fef5cbe0456e8f0dc81028a99904, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/1290c19e99684bdcbb3f4a593c4e6f55, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/310e728ef90c40fabdcae8c6da2168c6, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/f529d112ceb34b4daf65b21dfbfa8eb2] to archive 2023-04-08 20:57:06,378 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 20:57:06,380 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/2563655cd3564c919f057633491c42c4 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/2563655cd3564c919f057633491c42c4 2023-04-08 20:57:06,381 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/44a2955b55dc48a08a27f3dfe2f32859 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/44a2955b55dc48a08a27f3dfe2f32859 2023-04-08 20:57:06,382 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/965b8359c91e4630ba7366f7c2032b2f to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/965b8359c91e4630ba7366f7c2032b2f 2023-04-08 20:57:06,383 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/bbf254924bdf4b49a8db43108a5eee2f to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/bbf254924bdf4b49a8db43108a5eee2f 2023-04-08 20:57:06,384 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/a44f51f2b80c4c898cfe5af94538c04c to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/a44f51f2b80c4c898cfe5af94538c04c 2023-04-08 20:57:06,384 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/fdd43aa340584c03a1ecdc811a9c63fb to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/fdd43aa340584c03a1ecdc811a9c63fb 2023-04-08 20:57:06,385 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/602ad3539f16461fbd65ab93e087535c to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/602ad3539f16461fbd65ab93e087535c 2023-04-08 20:57:06,386 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/58c388679c3b46d69c5df366bec67c7f to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/58c388679c3b46d69c5df366bec67c7f 2023-04-08 20:57:06,387 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/c3bfa8c5b4c44c408738cf084b5e0770 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/c3bfa8c5b4c44c408738cf084b5e0770 2023-04-08 20:57:06,387 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/f082fef5cbe0456e8f0dc81028a99904 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/f082fef5cbe0456e8f0dc81028a99904 2023-04-08 20:57:06,389 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/1290c19e99684bdcbb3f4a593c4e6f55 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/1290c19e99684bdcbb3f4a593c4e6f55 2023-04-08 20:57:06,390 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/310e728ef90c40fabdcae8c6da2168c6 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/310e728ef90c40fabdcae8c6da2168c6 2023-04-08 20:57:06,391 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/f529d112ceb34b4daf65b21dfbfa8eb2 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/f529d112ceb34b4daf65b21dfbfa8eb2 2023-04-08 20:57:06,393 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/8a302cb8e9104fa7a19d469e31e2316d, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/9a5f54890148437885355dc1c3c9b2ac, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/7fc987c4d213417bbb894481dcffe159, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/0a688d4ae27a4c0185ab6105c5893415, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/480ebefa348d459cb4b826f360134ae5, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/818a54bbaf484dca92c437bc61db2831, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/494d814ab3ae4f6da2623e71a00f48bf, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/8b3b2580a6d4455182a60a9a5d6dc980, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/60f4ccf08ff8482fa9ec2938e130db47, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/b3d8515fd1f44bfd8a22131437d7aab5, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/06e620299d8849668d851122b1ec26e3, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/aea3be35b335475392ea1b4cada0b1c6, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/0a62d1c91ad24d30841531ea3d84b4ec] to archive 2023-04-08 20:57:06,394 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 20:57:06,395 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/8a302cb8e9104fa7a19d469e31e2316d to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/8a302cb8e9104fa7a19d469e31e2316d 2023-04-08 20:57:06,396 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/9a5f54890148437885355dc1c3c9b2ac to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/9a5f54890148437885355dc1c3c9b2ac 2023-04-08 20:57:06,397 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/7fc987c4d213417bbb894481dcffe159 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/7fc987c4d213417bbb894481dcffe159 2023-04-08 20:57:06,399 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/0a688d4ae27a4c0185ab6105c5893415 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/0a688d4ae27a4c0185ab6105c5893415 2023-04-08 20:57:06,400 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/480ebefa348d459cb4b826f360134ae5 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/480ebefa348d459cb4b826f360134ae5 2023-04-08 20:57:06,401 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/818a54bbaf484dca92c437bc61db2831 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/818a54bbaf484dca92c437bc61db2831 2023-04-08 20:57:06,402 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/494d814ab3ae4f6da2623e71a00f48bf to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/494d814ab3ae4f6da2623e71a00f48bf 2023-04-08 20:57:06,403 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/8b3b2580a6d4455182a60a9a5d6dc980 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/8b3b2580a6d4455182a60a9a5d6dc980 2023-04-08 20:57:06,404 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/60f4ccf08ff8482fa9ec2938e130db47 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/60f4ccf08ff8482fa9ec2938e130db47 2023-04-08 20:57:06,405 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/b3d8515fd1f44bfd8a22131437d7aab5 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/b3d8515fd1f44bfd8a22131437d7aab5 2023-04-08 20:57:06,407 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/06e620299d8849668d851122b1ec26e3 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/06e620299d8849668d851122b1ec26e3 2023-04-08 20:57:06,408 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/aea3be35b335475392ea1b4cada0b1c6 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/aea3be35b335475392ea1b4cada0b1c6 2023-04-08 20:57:06,409 DEBUG [StoreCloser-TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/0a62d1c91ad24d30841531ea3d84b4ec to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/0a62d1c91ad24d30841531ea3d84b4ec 2023-04-08 20:57:06,414 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/recovered.edits/247.seqid, newMaxSeqId=247, maxSeqId=4 2023-04-08 20:57:06,415 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc. 2023-04-08 20:57:06,415 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for eb4c5ed1339ded9a0ba1af8980e65ffc: 2023-04-08 20:57:06,416 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:57:06,416 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=eb4c5ed1339ded9a0ba1af8980e65ffc, regionState=CLOSED 2023-04-08 20:57:06,417 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987426416"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680987426416"}]},"ts":"1680987426416"} 2023-04-08 20:57:06,418 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-08 20:57:06,418 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure eb4c5ed1339ded9a0ba1af8980e65ffc, server=jenkins-hbase20.apache.org,41091,1680987313601 in 2.6410 sec 2023-04-08 20:57:06,419 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-08 20:57:06,419 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=eb4c5ed1339ded9a0ba1af8980e65ffc, UNASSIGN in 2.6450 sec 2023-04-08 20:57:06,420 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680987426420"}]},"ts":"1680987426420"} 2023-04-08 20:57:06,420 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-08 20:57:06,422 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-08 20:57:06,423 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.6550 sec 2023-04-08 20:57:07,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-08 20:57:07,880 INFO [Listener at localhost.localdomain/41823] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-08 20:57:07,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-08 20:57:07,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 20:57:07,883 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 20:57:07,884 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 20:57:07,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-08 20:57:07,892 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:57:07,896 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A, FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B, FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C, FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/recovered.edits] 2023-04-08 20:57:07,902 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/1b7ecf1a0da740f992a9ad06c55a705d to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/1b7ecf1a0da740f992a9ad06c55a705d 2023-04-08 20:57:07,904 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/d3e2dca96ace419cb1fbb1550d66a188 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/d3e2dca96ace419cb1fbb1550d66a188 2023-04-08 20:57:07,906 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/ee34c4af59f440b7b114487b276d4657 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/ee34c4af59f440b7b114487b276d4657 2023-04-08 20:57:07,907 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/f4b248bd8c074d69a31e8bb04de65570 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/A/f4b248bd8c074d69a31e8bb04de65570 2023-04-08 20:57:07,910 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/1fedfa15cabf4aa881355ec6457b18e7 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/1fedfa15cabf4aa881355ec6457b18e7 2023-04-08 20:57:07,911 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/240c38604580442bbb8d1fc9a28614be to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/240c38604580442bbb8d1fc9a28614be 2023-04-08 20:57:07,912 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/40b02c4c54674aa5a4368e0db8142d94 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/40b02c4c54674aa5a4368e0db8142d94 2023-04-08 20:57:07,913 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/b6985a1222d94370b70c90e5fb95f3a9 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/B/b6985a1222d94370b70c90e5fb95f3a9 2023-04-08 20:57:07,915 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/4431b1998c45423f95bcb89625347372 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/4431b1998c45423f95bcb89625347372 2023-04-08 20:57:07,916 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/48da29b3f20742e48ccd187943f3dc6d to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/48da29b3f20742e48ccd187943f3dc6d 2023-04-08 20:57:07,917 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/79ab0b2cc7114473b3ffbffdde33cc3c to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/79ab0b2cc7114473b3ffbffdde33cc3c 2023-04-08 20:57:07,918 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/aabd594fcf6a4776898cb15eb4009ed5 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/C/aabd594fcf6a4776898cb15eb4009ed5 2023-04-08 20:57:07,920 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/recovered.edits/247.seqid to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc/recovered.edits/247.seqid 2023-04-08 20:57:07,921 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/eb4c5ed1339ded9a0ba1af8980e65ffc 2023-04-08 20:57:07,921 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-08 20:57:07,921 DEBUG [PEWorker-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-08 20:57:07,922 DEBUG [PEWorker-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-08 20:57:07,924 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304080673231aedeb4fbc8ac73a9cab6aaeaf to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304080673231aedeb4fbc8ac73a9cab6aaeaf 2023-04-08 20:57:07,925 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304081ce2da41d8734a3d8e4e0522c3a33ecf to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304081ce2da41d8734a3d8e4e0522c3a33ecf 2023-04-08 20:57:07,926 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304085d9b5db4236642ca952f5bbba3cf3e42 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304085d9b5db4236642ca952f5bbba3cf3e42 2023-04-08 20:57:07,927 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304085fa758a6331e4c5199027394041fa4f3 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304085fa758a6331e4c5199027394041fa4f3 2023-04-08 20:57:07,933 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304086fa79621714c4fbb925e64ba63b61028 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304086fa79621714c4fbb925e64ba63b61028 2023-04-08 20:57:07,934 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304087c0c1228de324ced87aff08724079fa1 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304087c0c1228de324ced87aff08724079fa1 2023-04-08 20:57:07,935 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040885f5e005ea864e7baeb69b9171543fd7 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040885f5e005ea864e7baeb69b9171543fd7 2023-04-08 20:57:07,936 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408956fc13a237049748ee5267ad3745453 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408956fc13a237049748ee5267ad3745453 2023-04-08 20:57:07,936 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304089e69e58721a54ffb8d5765e0e791fe6d to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304089e69e58721a54ffb8d5765e0e791fe6d 2023-04-08 20:57:07,937 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408b63b5497689a4dafb3ad8a805ca4dd32 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408b63b5497689a4dafb3ad8a805ca4dd32 2023-04-08 20:57:07,938 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408b83efbed8af649f281ada019bb40ee81 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408b83efbed8af649f281ada019bb40ee81 2023-04-08 20:57:07,939 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408deb0b5b44fad4609a1b0b39d126de98d to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408deb0b5b44fad4609a1b0b39d126de98d 2023-04-08 20:57:07,939 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408e5703a2fa8614f1f991d90487f5ad0f9 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408e5703a2fa8614f1f991d90487f5ad0f9 2023-04-08 20:57:07,940 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408e9c7f68cc658458ab29b2c5fb9c2e9db to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408e9c7f68cc658458ab29b2c5fb9c2e9db 2023-04-08 20:57:07,941 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408f3ac260347a94d6aa6b767c933916a87 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408f3ac260347a94d6aa6b767c933916a87 2023-04-08 20:57:07,941 DEBUG [PEWorker-4] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-08 20:57:07,943 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 20:57:07,944 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-08 20:57:07,945 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-08 20:57:07,946 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 20:57:07,946 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-08 20:57:07,946 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680987427946"}]},"ts":"9223372036854775807"} 2023-04-08 20:57:07,948 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-08 20:57:07,948 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => eb4c5ed1339ded9a0ba1af8980e65ffc, NAME => 'TestAcidGuarantees,,1680987400757.eb4c5ed1339ded9a0ba1af8980e65ffc.', STARTKEY => '', ENDKEY => ''}] 2023-04-08 20:57:07,948 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-08 20:57:07,948 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680987427948"}]},"ts":"9223372036854775807"} 2023-04-08 20:57:07,949 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-08 20:57:07,951 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 20:57:07,951 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 70 msec 2023-04-08 20:57:07,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-08 20:57:07,986 INFO [Listener at localhost.localdomain/41823] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-08 20:57:07,999 INFO [Listener at localhost.localdomain/41823] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=227 (was 229), OpenFileDescriptor=511 (was 531), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=358 (was 367), ProcessCount=166 (was 166), AvailableMemoryMB=3510 (was 3459) - AvailableMemoryMB LEAK? - 2023-04-08 20:57:08,011 INFO [Listener at localhost.localdomain/41823] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=227, OpenFileDescriptor=511, MaxFileDescriptor=60000, SystemLoadAverage=358, ProcessCount=166, AvailableMemoryMB=3510 2023-04-08 20:57:08,013 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-08 20:57:08,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-08 20:57:08,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-08 20:57:08,016 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-08 20:57:08,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-08 20:57:08,016 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-08 20:57:08,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-08 20:57:08,017 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be 2023-04-08 20:57:08,018 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be empty. 2023-04-08 20:57:08,018 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be 2023-04-08 20:57:08,018 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-08 20:57:08,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-08 20:57:08,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-08 20:57:08,428 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-08 20:57:08,431 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 0013c9607e8a284ca05e07f23e7866be, NAME => 'TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp 2023-04-08 20:57:08,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-08 20:57:08,839 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 20:57:08,839 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 0013c9607e8a284ca05e07f23e7866be, disabling compactions & flushes 2023-04-08 20:57:08,839 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:08,839 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:08,839 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. after waiting 0 ms 2023-04-08 20:57:08,839 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:08,839 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:08,839 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:08,844 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-08 20:57:08,846 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987428845"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680987428845"}]},"ts":"1680987428845"} 2023-04-08 20:57:08,848 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-08 20:57:08,849 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-08 20:57:08,849 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680987428849"}]},"ts":"1680987428849"} 2023-04-08 20:57:08,851 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-08 20:57:08,856 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0013c9607e8a284ca05e07f23e7866be, ASSIGN}] 2023-04-08 20:57:08,859 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0013c9607e8a284ca05e07f23e7866be, ASSIGN 2023-04-08 20:57:08,860 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=0013c9607e8a284ca05e07f23e7866be, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,41091,1680987313601; forceNewPlan=false, retain=false 2023-04-08 20:57:09,011 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=0013c9607e8a284ca05e07f23e7866be, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:09,012 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987429011"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680987429011"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680987429011"}]},"ts":"1680987429011"} 2023-04-08 20:57:09,013 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601}] 2023-04-08 20:57:09,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-08 20:57:09,174 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:09,175 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 0013c9607e8a284ca05e07f23e7866be, NAME => 'TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.', STARTKEY => '', ENDKEY => ''} 2023-04-08 20:57:09,175 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 0013c9607e8a284ca05e07f23e7866be 2023-04-08 20:57:09,176 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 20:57:09,176 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 0013c9607e8a284ca05e07f23e7866be 2023-04-08 20:57:09,176 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 0013c9607e8a284ca05e07f23e7866be 2023-04-08 20:57:09,178 INFO [StoreOpener-0013c9607e8a284ca05e07f23e7866be-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 0013c9607e8a284ca05e07f23e7866be 2023-04-08 20:57:09,179 DEBUG [StoreOpener-0013c9607e8a284ca05e07f23e7866be-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A 2023-04-08 20:57:09,179 DEBUG [StoreOpener-0013c9607e8a284ca05e07f23e7866be-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A 2023-04-08 20:57:09,179 INFO [StoreOpener-0013c9607e8a284ca05e07f23e7866be-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 20:57:09,180 INFO [StoreOpener-0013c9607e8a284ca05e07f23e7866be-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0013c9607e8a284ca05e07f23e7866be columnFamilyName A 2023-04-08 20:57:09,180 INFO [StoreOpener-0013c9607e8a284ca05e07f23e7866be-1] regionserver.HStore(310): Store=0013c9607e8a284ca05e07f23e7866be/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 20:57:09,180 INFO [StoreOpener-0013c9607e8a284ca05e07f23e7866be-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 0013c9607e8a284ca05e07f23e7866be 2023-04-08 20:57:09,182 DEBUG [StoreOpener-0013c9607e8a284ca05e07f23e7866be-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B 2023-04-08 20:57:09,182 DEBUG [StoreOpener-0013c9607e8a284ca05e07f23e7866be-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B 2023-04-08 20:57:09,182 INFO [StoreOpener-0013c9607e8a284ca05e07f23e7866be-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 20:57:09,182 INFO [StoreOpener-0013c9607e8a284ca05e07f23e7866be-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0013c9607e8a284ca05e07f23e7866be columnFamilyName B 2023-04-08 20:57:09,183 INFO [StoreOpener-0013c9607e8a284ca05e07f23e7866be-1] regionserver.HStore(310): Store=0013c9607e8a284ca05e07f23e7866be/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 20:57:09,183 INFO [StoreOpener-0013c9607e8a284ca05e07f23e7866be-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 0013c9607e8a284ca05e07f23e7866be 2023-04-08 20:57:09,184 DEBUG [StoreOpener-0013c9607e8a284ca05e07f23e7866be-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C 2023-04-08 20:57:09,184 DEBUG [StoreOpener-0013c9607e8a284ca05e07f23e7866be-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C 2023-04-08 20:57:09,184 INFO [StoreOpener-0013c9607e8a284ca05e07f23e7866be-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 20:57:09,185 INFO [StoreOpener-0013c9607e8a284ca05e07f23e7866be-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0013c9607e8a284ca05e07f23e7866be columnFamilyName C 2023-04-08 20:57:09,185 INFO [StoreOpener-0013c9607e8a284ca05e07f23e7866be-1] regionserver.HStore(310): Store=0013c9607e8a284ca05e07f23e7866be/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 20:57:09,185 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:09,186 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be 2023-04-08 20:57:09,186 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be 2023-04-08 20:57:09,188 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-08 20:57:09,189 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 0013c9607e8a284ca05e07f23e7866be 2023-04-08 20:57:09,191 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-08 20:57:09,192 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 0013c9607e8a284ca05e07f23e7866be; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61129549, jitterRate=-0.0890987366437912}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-08 20:57:09,192 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:09,193 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be., pid=49, masterSystemTime=1680987429165 2023-04-08 20:57:09,194 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:09,194 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:09,194 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=0013c9607e8a284ca05e07f23e7866be, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:09,195 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987429194"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680987429194"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680987429194"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680987429194"}]},"ts":"1680987429194"} 2023-04-08 20:57:09,197 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-08 20:57:09,197 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 in 183 msec 2023-04-08 20:57:09,199 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-08 20:57:09,199 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=0013c9607e8a284ca05e07f23e7866be, ASSIGN in 341 msec 2023-04-08 20:57:09,200 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-08 20:57:09,200 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680987429200"}]},"ts":"1680987429200"} 2023-04-08 20:57:09,201 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-08 20:57:09,203 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-08 20:57:09,205 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1900 sec 2023-04-08 20:57:10,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-08 20:57:10,127 INFO [Listener at localhost.localdomain/41823] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-08 20:57:10,129 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x7cb49278 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:57:10,139 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4df55fe5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:57:10,142 DEBUG [Listener at localhost.localdomain/41823] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:57:10,146 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60746, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:57:10,148 DEBUG [Listener at localhost.localdomain/41823] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-08 20:57:10,151 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41160, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-08 20:57:10,153 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x50e89dd2 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:57:10,158 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@817e2e5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:57:10,159 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x545adde0 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:57:10,166 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4048dcb4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:57:10,167 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x4c4d3f2e to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:57:10,172 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4e7a5e9a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:57:10,173 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x47d58a9b to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:57:10,177 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@62447900, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:57:10,177 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x3af76165 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:57:10,181 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@519db5f2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:57:10,181 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x5c9c1feb to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:57:10,185 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@609213af, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:57:10,185 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x22bf9ddd to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:57:10,198 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1ee0ad26, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:57:10,199 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x34c4ae9a to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:57:10,207 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6acb8caa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:57:10,208 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x6187ef67 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:57:10,216 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70263db3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:57:10,216 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x7bf0ef8b to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:57:10,223 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@25d2db45, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:57:10,231 DEBUG [hconnection-0x72e57f99-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:57:10,232 DEBUG [hconnection-0x65b49bbe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:57:10,232 DEBUG [hconnection-0x7d8abf99-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:57:10,232 DEBUG [hconnection-0x4690c03b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:57:10,236 DEBUG [hconnection-0x5ea330ec-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:57:10,237 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60760, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:57:10,237 DEBUG [hconnection-0x754481c0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:57:10,237 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60764, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:57:10,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:10,238 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60798, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:57:10,239 DEBUG [hconnection-0x9b63730-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:57:10,245 DEBUG [hconnection-0xbe98ecd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:57:10,245 DEBUG [hconnection-0x57e67624-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:57:10,247 DEBUG [hconnection-0x26288ce3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:57:10,248 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60788, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:57:10,248 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60772, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:57:10,251 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60808, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:57:10,251 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60804, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:57:10,252 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60824, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:57:10,256 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60830, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:57:10,256 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60828, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:57:10,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:10,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:10,266 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:10,266 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:10,266 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:10,266 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:10,267 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:10,267 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:10,268 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:10,268 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:10,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:10,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:10,268 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:10,268 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:10,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:10,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:10,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:10,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:10,269 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:10,270 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:10,270 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:10,270 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:10,270 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:10,270 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:10,271 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:10,271 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:10,271 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:10,271 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:10,272 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0013c9607e8a284ca05e07f23e7866be 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-08 20:57:10,273 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=A 2023-04-08 20:57:10,273 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:10,273 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=B 2023-04-08 20:57:10,273 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:10,273 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=C 2023-04-08 20:57:10,273 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:10,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 0013c9607e8a284ca05e07f23e7866be 2023-04-08 20:57:10,277 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:10,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680987430282, max timestamp=1680987430289Num uniques 859; 2023-04-08 20:57:10,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-08 20:57:10,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680987430282, max timestamp=1680987430289Num uniques 859; 2023-04-08 20:57:10,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-08 20:57:10,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680987430282, max timestamp=1680987430289Num uniques 859; 2023-04-08 20:57:10,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-08 20:57:10,299 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:10,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60824 deadline: 1680987490298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:10,299 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:10,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:10,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60788 deadline: 1680987490298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:10,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60798 deadline: 1680987490298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:10,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680987430296, max timestamp=1680987430298Num uniques 1168; 2023-04-08 20:57:10,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=12268, suffix heap size=149440,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=268 2023-04-08 20:57:10,301 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680987430296, max timestamp=1680987430298Num uniques 1168; 2023-04-08 20:57:10,301 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=12268, suffix heap size=149440,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=268 2023-04-08 20:57:10,301 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680987430296, max timestamp=1680987430298Num uniques 1168; 2023-04-08 20:57:10,301 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=12268, suffix heap size=149440,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=268 2023-04-08 20:57:10,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:10,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60828 deadline: 1680987490320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:10,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:10,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60764 deadline: 1680987490322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:10,404 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:10,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60788 deadline: 1680987490400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:10,405 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:10,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60798 deadline: 1680987490400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:10,405 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:10,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60824 deadline: 1680987490401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:10,426 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:10,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60764 deadline: 1680987490423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:10,426 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:10,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60828 deadline: 1680987490423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:10,608 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:10,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60798 deadline: 1680987490606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:10,609 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:10,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60788 deadline: 1680987490606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:10,609 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:10,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60824 deadline: 1680987490606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:10,630 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:10,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60764 deadline: 1680987490628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:10,630 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:10,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60828 deadline: 1680987490628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:10,709 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/03f61d02f4924fae8c9aa89ae7de19f2 2023-04-08 20:57:10,718 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/e13e6aaa23064e3a8a3eb2c30c173178 2023-04-08 20:57:10,913 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:10,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60824 deadline: 1680987490910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:10,914 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:10,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60798 deadline: 1680987490910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:10,916 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:10,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60788 deadline: 1680987490912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:10,934 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:10,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60828 deadline: 1680987490931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:10,934 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:10,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60764 deadline: 1680987490932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:11,126 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/6009172b69ef4aca95bf19b50cd7ad7d 2023-04-08 20:57:11,132 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/03f61d02f4924fae8c9aa89ae7de19f2 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/03f61d02f4924fae8c9aa89ae7de19f2 2023-04-08 20:57:11,135 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/03f61d02f4924fae8c9aa89ae7de19f2, entries=200, sequenceid=9, filesize=13.9 K 2023-04-08 20:57:11,136 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/e13e6aaa23064e3a8a3eb2c30c173178 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/e13e6aaa23064e3a8a3eb2c30c173178 2023-04-08 20:57:11,139 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/e13e6aaa23064e3a8a3eb2c30c173178, entries=150, sequenceid=9, filesize=11.6 K 2023-04-08 20:57:11,139 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/6009172b69ef4aca95bf19b50cd7ad7d as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/6009172b69ef4aca95bf19b50cd7ad7d 2023-04-08 20:57:11,142 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/6009172b69ef4aca95bf19b50cd7ad7d, entries=150, sequenceid=9, filesize=11.6 K 2023-04-08 20:57:11,143 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 0013c9607e8a284ca05e07f23e7866be in 871ms, sequenceid=9, compaction requested=false 2023-04-08 20:57:11,143 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:11,143 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:11,143 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:11,143 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:11,143 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:11,143 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:11,143 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:11,144 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:11,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:11,144 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:11,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:11,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:11,144 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:11,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:11,144 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:11,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:11,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:11,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:11,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:11,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:11,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b78b2f0[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:11,145 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:11,145 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:11,146 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:11,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:11,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:11,146 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:11,146 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:11,146 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:11,146 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:11,146 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:11,147 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:11,147 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:11,147 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:11,147 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:11,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:11,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:11,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:11,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:11,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:11,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:11,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:11,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:11,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:11,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:11,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:11,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:11,150 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:11,150 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:11,150 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:11,150 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:11,150 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:11,151 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:11,151 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:11,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:11,151 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:11,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:11,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:11,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:11,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:11,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:11,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:11,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:11,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:11,152 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:11,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:11,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:11,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:11,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:11,154 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:11,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:11,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:11,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:11,155 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:11,155 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:11,155 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:11,155 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:11,155 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:11,155 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:11,155 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:11,155 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:11,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:11,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:11,156 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:11,156 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:11,156 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:11,156 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:11,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:11,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:11,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:11,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:11,256 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:11,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:11,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:11,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:11,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:11,258 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:11,258 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:11,258 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:11,258 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:11,259 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:11,259 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:11,259 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:11,259 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:11,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:11,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:11,260 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:11,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:11,260 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:11,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:11,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:11,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:11,260 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:11,261 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:11,261 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:11,261 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:11,261 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:11,261 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:11,261 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:11,261 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:11,261 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:11,261 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:11,261 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0013c9607e8a284ca05e07f23e7866be 3/3 column families, dataSize=147.60 KB heapSize=414.61 KB 2023-04-08 20:57:11,261 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=A 2023-04-08 20:57:11,261 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:57:11,262 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=B 2023-04-08 20:57:11,262 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:57:11,262 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=C 2023-04-08 20:57:11,262 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:57:11,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 0013c9607e8a284ca05e07f23e7866be 2023-04-08 20:57:11,420 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:11,428 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:11,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60788 deadline: 1680987491422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:11,428 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:11,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60798 deadline: 1680987491423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:11,435 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:11,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60824 deadline: 1680987491429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:11,442 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:11,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60828 deadline: 1680987491439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:11,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:11,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60764 deadline: 1680987491439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:11,531 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:11,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60788 deadline: 1680987491529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:11,531 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:11,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60798 deadline: 1680987491529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:11,538 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:11,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60824 deadline: 1680987491536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:11,680 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/54b42cda3aa7495faf6a5c98b5a78c37 2023-04-08 20:57:11,734 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:11,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60798 deadline: 1680987491732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:11,735 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:11,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60788 deadline: 1680987491733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:11,743 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:11,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60824 deadline: 1680987491740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:11,943 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-08 20:57:12,043 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:12,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60798 deadline: 1680987492036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,043 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:12,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60788 deadline: 1680987492037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,048 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:12,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60824 deadline: 1680987492044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,089 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/1b2e9343bfa540c2ab3f40bede8050d5 2023-04-08 20:57:12,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:12,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60828 deadline: 1680987492446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,456 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:12,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60764 deadline: 1680987492453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,498 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/266288b8ad944bada79aca03fe1cea3f 2023-04-08 20:57:12,501 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/54b42cda3aa7495faf6a5c98b5a78c37 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/54b42cda3aa7495faf6a5c98b5a78c37 2023-04-08 20:57:12,505 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/54b42cda3aa7495faf6a5c98b5a78c37, entries=150, sequenceid=63, filesize=11.6 K 2023-04-08 20:57:12,505 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/1b2e9343bfa540c2ab3f40bede8050d5 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/1b2e9343bfa540c2ab3f40bede8050d5 2023-04-08 20:57:12,508 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/1b2e9343bfa540c2ab3f40bede8050d5, entries=150, sequenceid=63, filesize=11.6 K 2023-04-08 20:57:12,509 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/266288b8ad944bada79aca03fe1cea3f as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/266288b8ad944bada79aca03fe1cea3f 2023-04-08 20:57:12,512 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/266288b8ad944bada79aca03fe1cea3f, entries=150, sequenceid=63, filesize=11.6 K 2023-04-08 20:57:12,512 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~414.56 KB/424512, currentSize=33.54 KB/34350 for 0013c9607e8a284ca05e07f23e7866be in 1251ms, sequenceid=63, compaction requested=false 2023-04-08 20:57:12,512 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:12,512 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:12,512 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:12,512 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:12,512 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,513 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:12,513 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:12,513 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,513 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:12,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:12,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:12,514 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:12,514 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:12,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:12,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:12,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:12,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:12,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:12,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6443869d[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:12,515 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:12,515 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:12,515 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:12,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:12,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:12,515 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:12,515 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:12,515 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:12,516 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,516 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:12,517 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,517 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:12,517 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:12,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:12,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:12,517 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:12,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:12,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:12,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:12,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:12,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:12,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:12,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:12,519 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:12,519 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:12,519 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:12,519 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:12,519 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:12,520 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:12,520 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:12,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:12,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:12,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:12,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:12,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:12,520 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:12,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:12,521 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:12,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:12,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:12,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:12,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:12,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:12,522 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:12,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:12,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,524 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,524 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:12,524 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:12,524 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:12,524 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:12,524 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:12,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:12,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:12,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:12,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:12,524 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:12,524 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:12,524 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:12,524 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:12,524 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:12,524 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:12,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:12,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:12,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 0013c9607e8a284ca05e07f23e7866be 2023-04-08 20:57:12,549 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0013c9607e8a284ca05e07f23e7866be 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-08 20:57:12,550 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=A 2023-04-08 20:57:12,550 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:12,550 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=B 2023-04-08 20:57:12,550 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:12,550 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=C 2023-04-08 20:57:12,550 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:12,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680987432578, max timestamp=1680987432609Num uniques 859; 2023-04-08 20:57:12,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-08 20:57:12,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680987432578, max timestamp=1680987432609Num uniques 859; 2023-04-08 20:57:12,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-08 20:57:12,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680987432578, max timestamp=1680987432609Num uniques 859; 2023-04-08 20:57:12,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-08 20:57:12,624 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:12,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:12,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:12,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:12,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:12,627 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:12,627 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:12,627 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:12,627 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:12,628 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:12,628 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:12,628 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,628 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:12,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:12,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:12,632 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:12,632 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:12,632 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:12,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:12,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:12,633 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:12,633 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:12,633 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:12,633 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:12,633 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:12,633 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:12,633 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:12,633 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:12,633 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:12,633 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:12,633 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:12,633 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:12,634 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:12,634 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:12,634 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,634 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:12,634 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:12,634 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:12,634 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:12,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:12,635 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:12,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:12,635 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:12,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:12,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:12,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:12,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:12,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37669dc6[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:12,636 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:12,636 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:12,637 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:12,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:12,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:12,637 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:12,637 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:12,637 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:12,637 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,637 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:12,638 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,638 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:12,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:12,638 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:12,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:12,638 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:12,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:12,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:12,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:12,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:12,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:12,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:12,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:12,640 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:12,640 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:12,640 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:12,640 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:12,640 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:12,640 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:12,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:12,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:12,641 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:12,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:12,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:12,641 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:12,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:12,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:12,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:12,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:12,641 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:12,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:12,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:12,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:12,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,642 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:12,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:12,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,643 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:12,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:12,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:12,644 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:12,644 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,644 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:12,644 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:12,644 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:12,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:12,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:12,644 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:12,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:12,644 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:12,644 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:12,644 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:12,644 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:12,644 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:12,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:12,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60798 deadline: 1680987492645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,660 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:12,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60788 deadline: 1680987492651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:12,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60824 deadline: 1680987492651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,744 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:12,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:12,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:12,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:12,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:12,747 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:12,747 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:12,747 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:12,747 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:12,747 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:12,747 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:12,748 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,748 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:12,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:12,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:12,748 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,748 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:12,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:12,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:12,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:12,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:12,749 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:12,749 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:12,749 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:12,749 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:12,749 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:12,749 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:12,749 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:12,749 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:12,749 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:12,749 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:12,749 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:12,749 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:12,749 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:12,750 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:12,750 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,750 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:12,750 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:12,750 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,750 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:12,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:12,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:12,750 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:12,751 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:12,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:12,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:12,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:12,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:12,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:12,752 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:12,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3689ed66[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:12,752 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:12,752 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:12,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:12,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:12,752 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:12,752 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:12,752 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,752 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:12,752 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:12,753 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,753 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:12,753 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:12,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:12,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:12,753 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:12,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:12,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:12,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:12,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:12,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:12,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:12,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:12,755 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:12,755 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:12,755 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:12,755 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:12,755 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:12,756 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:12,756 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:12,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:12,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:12,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:12,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:12,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:12,756 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:12,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:12,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:12,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:12,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:12,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:12,757 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:12,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,758 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,759 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:12,759 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,759 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:12,759 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:12,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:12,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:12,759 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:12,759 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:12,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:12,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:12,759 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,760 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:12,760 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:12,760 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:12,760 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:12,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:12,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:12,760 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:12,763 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:12,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60798 deadline: 1680987492760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,766 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:12,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60788 deadline: 1680987492761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,766 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:12,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60824 deadline: 1680987492761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,860 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:12,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:12,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:12,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:12,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:12,863 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:12,863 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:12,863 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:12,863 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:12,863 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:12,863 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:12,864 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,864 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,864 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:12,866 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:12,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:12,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:12,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:12,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:12,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:12,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:12,867 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:12,867 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:12,867 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:12,867 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:12,867 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:12,867 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:12,867 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:12,867 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:12,867 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:12,867 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:12,868 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:12,868 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:12,868 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:12,868 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:12,868 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,868 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:12,868 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:12,868 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:12,868 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:12,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:12,869 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:12,869 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:12,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:12,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:12,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:12,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:12,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:12,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68383675[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:12,870 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:12,870 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:12,870 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:12,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:12,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:12,871 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:12,871 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:12,871 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:12,871 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,871 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:12,871 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:12,871 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,871 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:12,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:12,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:12,871 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:12,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:12,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:12,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:12,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:12,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:12,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:12,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:12,873 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:12,873 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:12,873 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:12,873 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:12,873 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:12,874 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:12,874 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:12,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:12,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:12,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:12,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:12,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:12,874 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:12,874 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:12,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:12,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:12,875 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:12,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:12,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,877 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,877 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:12,877 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:12,878 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:12,878 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:12,878 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,878 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:12,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:12,878 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:12,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:12,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:12,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:12,878 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:12,878 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:12,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:12,878 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:12,878 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:12,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:12,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:12,966 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:12,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60798 deadline: 1680987492964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,972 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=74 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/239568da5d274a459d1ff4050c1558f1 2023-04-08 20:57:12,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:12,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60788 deadline: 1680987492968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:12,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60824 deadline: 1680987492968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,978 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:12,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:12,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:12,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:12,989 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=74 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/36f31aa3328a4a08bd5531e56f1527e5 2023-04-08 20:57:12,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:12,989 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:12,989 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:12,989 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:12,990 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:12,990 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:12,990 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:12,990 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,990 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:12,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:12,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:12,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:12,991 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,991 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:12,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:12,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:12,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:12,992 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:12,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:12,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:12,992 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:12,992 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:12,992 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:12,992 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:12,992 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:12,992 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:12,992 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:12,992 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:12,993 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:12,993 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:12,993 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:12,993 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,993 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:12,993 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:12,994 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,994 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:12,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:12,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:12,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:12,994 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:12,994 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:12,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:12,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:12,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:12,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:12,995 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:12,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10b72281[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:12,995 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:12,996 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:12,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:12,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:12,996 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:12,996 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:12,996 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:12,996 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,996 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:12,997 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,997 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:12,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:12,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:12,997 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:12,997 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:12,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:12,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:12,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:12,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:12,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:12,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:12,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:12,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:12,999 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:13,000 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:13,000 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:13,000 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:13,000 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:13,000 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:13,001 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:13,001 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:13,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:13,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:13,001 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:13,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:13,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:13,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:13,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:13,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:13,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:13,002 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:13,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:13,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:13,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:13,003 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:13,003 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:13,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:13,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:13,003 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,003 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:13,003 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:13,004 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:13,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:13,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:13,004 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,004 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:13,004 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:13,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:13,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:13,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:13,004 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:13,004 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:13,004 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:13,104 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:13,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:13,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:13,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:13,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:13,107 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:13,107 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:13,107 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:13,107 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:13,107 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:13,107 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:13,108 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:13,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:13,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:13,108 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:13,108 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,108 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:13,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:13,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:13,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:13,109 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:13,109 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:13,109 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:13,109 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:13,109 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:13,109 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:13,109 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:13,109 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:13,109 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:13,109 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:13,109 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:13,109 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:13,109 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:13,109 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:13,109 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,110 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:13,110 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:13,110 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,110 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:13,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:13,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:13,111 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:13,111 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:13,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:13,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:13,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:13,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:13,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:13,112 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:13,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13bc15fc[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:13,112 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:13,112 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:13,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:13,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:13,112 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:13,113 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:13,113 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:13,114 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,114 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:13,114 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,114 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:13,114 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:13,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:13,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:13,114 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:13,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:13,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:13,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:13,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:13,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:13,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:13,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:13,116 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:13,116 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:13,116 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:13,116 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:13,116 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:13,117 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:13,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:13,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:13,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:13,117 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:13,117 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:13,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:13,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:13,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:13,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:13,117 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:13,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:13,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:13,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:13,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:13,118 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:13,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:13,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,120 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,120 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:13,120 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:13,120 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:13,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:13,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:13,120 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:13,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:13,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:13,120 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:13,120 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,120 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:13,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:13,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:13,120 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:13,121 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:13,121 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:13,121 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:13,221 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:13,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:13,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:13,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:13,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:13,223 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:13,223 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:13,223 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:13,223 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:13,224 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:13,224 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:13,224 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,224 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:13,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:13,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:13,225 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,225 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:13,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:13,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:13,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:13,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:13,225 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:13,225 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:13,226 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:13,226 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:13,226 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:13,226 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:13,226 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:13,226 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:13,226 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:13,226 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:13,226 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:13,226 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:13,226 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:13,226 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,226 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:13,226 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:13,227 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:13,227 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:13,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:13,227 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:13,227 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:13,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:13,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:13,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:13,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:13,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:13,228 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:13,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a272830[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:13,228 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:13,229 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:13,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:13,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:13,229 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:13,229 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:13,229 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:13,229 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:13,229 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:13,229 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:13,229 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,230 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,230 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:13,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:13,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:13,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:13,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:13,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:13,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:13,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:13,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:13,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:13,232 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:13,232 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:13,232 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:13,232 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:13,232 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:13,232 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:13,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:13,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:13,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:13,233 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:13,233 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:13,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:13,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:13,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:13,233 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:13,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:13,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:13,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:13,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:13,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:13,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,234 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:13,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:13,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,236 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:13,236 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:13,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:13,236 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:13,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:13,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:13,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:13,236 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,236 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:13,236 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:13,236 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:13,236 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,236 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:13,236 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:13,236 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:13,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:13,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:13,237 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:13,273 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:13,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60798 deadline: 1680987493269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:13,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60824 deadline: 1680987493275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:13,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60788 deadline: 1680987493276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,337 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:13,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:13,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:13,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:13,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:13,339 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:13,339 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:13,340 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:13,340 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:13,340 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:13,340 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:13,341 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,341 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:13,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:13,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:13,341 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,341 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:13,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:13,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:13,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:13,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:13,341 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:13,342 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:13,342 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:13,342 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:13,342 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:13,342 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:13,342 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:13,342 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:13,342 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:13,342 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:13,342 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:13,342 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:13,342 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:13,342 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:13,342 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,342 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:13,342 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:13,343 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,343 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:13,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:13,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:13,343 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:13,343 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:13,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:13,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:13,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:13,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:13,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:13,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4305309f[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:13,345 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:13,345 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:13,346 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:13,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:13,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:13,346 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,346 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:13,346 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:13,346 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:13,346 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:13,347 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:13,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:13,347 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:13,347 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:13,347 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:13,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:13,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:13,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:13,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:13,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:13,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:13,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:13,349 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:13,349 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:13,349 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:13,349 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:13,349 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:13,350 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:13,350 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:13,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:13,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:13,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:13,350 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:13,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:13,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:13,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:13,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:13,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:13,350 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:13,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:13,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:13,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:13,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,351 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:13,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:13,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,352 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,352 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:13,353 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:13,352 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:13,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:13,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:13,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:13,353 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:13,353 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,353 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:13,353 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:13,353 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:13,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:13,353 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:13,353 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:13,353 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:13,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:13,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:13,400 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=74 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/676f1ca83f064f578320256d3b9202eb 2023-04-08 20:57:13,403 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/239568da5d274a459d1ff4050c1558f1 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/239568da5d274a459d1ff4050c1558f1 2023-04-08 20:57:13,406 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/239568da5d274a459d1ff4050c1558f1, entries=250, sequenceid=74, filesize=16.2 K 2023-04-08 20:57:13,406 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/36f31aa3328a4a08bd5531e56f1527e5 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/36f31aa3328a4a08bd5531e56f1527e5 2023-04-08 20:57:13,409 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/36f31aa3328a4a08bd5531e56f1527e5, entries=150, sequenceid=74, filesize=11.6 K 2023-04-08 20:57:13,409 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/676f1ca83f064f578320256d3b9202eb as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/676f1ca83f064f578320256d3b9202eb 2023-04-08 20:57:13,412 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/676f1ca83f064f578320256d3b9202eb, entries=150, sequenceid=74, filesize=11.6 K 2023-04-08 20:57:13,413 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 0013c9607e8a284ca05e07f23e7866be in 864ms, sequenceid=74, compaction requested=true 2023-04-08 20:57:13,413 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:13,413 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:57:13,413 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:57:13,413 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:57:13,413 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:57:13,414 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:57:13,414 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42678 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:57:13,414 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:57:13,414 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): 0013c9607e8a284ca05e07f23e7866be/A is initiating minor compaction (all files) 2023-04-08 20:57:13,414 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 0013c9607e8a284ca05e07f23e7866be/B is initiating minor compaction (all files) 2023-04-08 20:57:13,414 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0013c9607e8a284ca05e07f23e7866be/A in TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:13,414 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0013c9607e8a284ca05e07f23e7866be/B in TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:13,414 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/03f61d02f4924fae8c9aa89ae7de19f2, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/54b42cda3aa7495faf6a5c98b5a78c37, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/239568da5d274a459d1ff4050c1558f1] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp, totalSize=41.7 K 2023-04-08 20:57:13,414 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/e13e6aaa23064e3a8a3eb2c30c173178, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/1b2e9343bfa540c2ab3f40bede8050d5, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/36f31aa3328a4a08bd5531e56f1527e5] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp, totalSize=34.8 K 2023-04-08 20:57:13,415 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 03f61d02f4924fae8c9aa89ae7de19f2, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1680987430258 2023-04-08 20:57:13,415 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting e13e6aaa23064e3a8a3eb2c30c173178, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1680987430258 2023-04-08 20:57:13,415 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 54b42cda3aa7495faf6a5c98b5a78c37, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1680987430318 2023-04-08 20:57:13,415 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 1b2e9343bfa540c2ab3f40bede8050d5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1680987430318 2023-04-08 20:57:13,415 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 239568da5d274a459d1ff4050c1558f1, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=74, earliestPutTs=1680987431420 2023-04-08 20:57:13,415 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 36f31aa3328a4a08bd5531e56f1527e5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=74, earliestPutTs=1680987432544 2023-04-08 20:57:13,420 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 0013c9607e8a284ca05e07f23e7866be#B#compaction#362 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:57:13,421 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0013c9607e8a284ca05e07f23e7866be#A#compaction#361 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:57:13,430 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/8e508f63691e4e558db0f7a83d2f832a as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/8e508f63691e4e558db0f7a83d2f832a 2023-04-08 20:57:13,434 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0013c9607e8a284ca05e07f23e7866be/A of 0013c9607e8a284ca05e07f23e7866be into 8e508f63691e4e558db0f7a83d2f832a(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:57:13,434 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:13,434 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be., storeName=0013c9607e8a284ca05e07f23e7866be/A, priority=13, startTime=1680987433413; duration=0sec 2023-04-08 20:57:13,434 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:57:13,434 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:57:13,435 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:57:13,435 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): 0013c9607e8a284ca05e07f23e7866be/C is initiating minor compaction (all files) 2023-04-08 20:57:13,435 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0013c9607e8a284ca05e07f23e7866be/C in TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:13,435 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/6009172b69ef4aca95bf19b50cd7ad7d, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/266288b8ad944bada79aca03fe1cea3f, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/676f1ca83f064f578320256d3b9202eb] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp, totalSize=34.8 K 2023-04-08 20:57:13,435 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 6009172b69ef4aca95bf19b50cd7ad7d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1680987430258 2023-04-08 20:57:13,436 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 266288b8ad944bada79aca03fe1cea3f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1680987430318 2023-04-08 20:57:13,436 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 676f1ca83f064f578320256d3b9202eb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=74, earliestPutTs=1680987432544 2023-04-08 20:57:13,441 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0013c9607e8a284ca05e07f23e7866be#C#compaction#363 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:57:13,453 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:13,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:13,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:13,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:13,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:13,456 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:13,456 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:13,456 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:13,456 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:13,456 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:13,456 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:13,457 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,457 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:13,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:13,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:13,457 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,457 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:13,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:13,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:13,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:13,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:13,458 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:13,459 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:13,459 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:13,459 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:13,459 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:13,459 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:13,459 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:13,459 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:13,459 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:13,459 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:13,459 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0013c9607e8a284ca05e07f23e7866be 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-08 20:57:13,460 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=A 2023-04-08 20:57:13,460 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:57:13,460 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=B 2023-04-08 20:57:13,460 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:57:13,460 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=C 2023-04-08 20:57:13,460 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:57:13,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 0013c9607e8a284ca05e07f23e7866be 2023-04-08 20:57:13,779 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:13,795 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:13,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60798 deadline: 1680987493791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:13,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60788 deadline: 1680987493794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:13,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60824 deadline: 1680987493795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,828 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/58e8e8f8b93a4a3292a08281b3fdd82a as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/58e8e8f8b93a4a3292a08281b3fdd82a 2023-04-08 20:57:13,833 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0013c9607e8a284ca05e07f23e7866be/B of 0013c9607e8a284ca05e07f23e7866be into 58e8e8f8b93a4a3292a08281b3fdd82a(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:57:13,833 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:13,833 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be., storeName=0013c9607e8a284ca05e07f23e7866be/B, priority=13, startTime=1680987433413; duration=0sec 2023-04-08 20:57:13,833 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:57:13,849 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/8a9d563c538941238460b51ebfafed8f as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/8a9d563c538941238460b51ebfafed8f 2023-04-08 20:57:13,852 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0013c9607e8a284ca05e07f23e7866be/C of 0013c9607e8a284ca05e07f23e7866be into 8a9d563c538941238460b51ebfafed8f(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:57:13,852 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:13,852 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be., storeName=0013c9607e8a284ca05e07f23e7866be/C, priority=13, startTime=1680987433414; duration=0sec 2023-04-08 20:57:13,852 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:57:13,874 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/d5330807be7240579d405c330845786a 2023-04-08 20:57:13,902 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:13,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60798 deadline: 1680987493896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,907 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:13,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60788 deadline: 1680987493905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:13,908 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:13,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60824 deadline: 1680987493905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:14,105 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:14,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60798 deadline: 1680987494104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:14,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:14,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60824 deadline: 1680987494108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:14,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:14,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60788 deadline: 1680987494109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:14,300 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/7bd446122bf546c89c0da304c2584963 2023-04-08 20:57:14,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:14,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60798 deadline: 1680987494407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:14,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:14,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60788 deadline: 1680987494412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:14,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:14,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60824 deadline: 1680987494413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:14,465 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:14,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60764 deadline: 1680987494462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:14,467 DEBUG [Thread-2751] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4143 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be., hostname=jenkins-hbase20.apache.org,41091,1680987313601, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 20:57:14,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:14,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60828 deadline: 1680987494467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:14,473 DEBUG [Thread-2745] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4151 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be., hostname=jenkins-hbase20.apache.org,41091,1680987313601, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 20:57:14,714 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/a915bdce531642a894e51bd7bb085b63 2023-04-08 20:57:14,718 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/d5330807be7240579d405c330845786a as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/d5330807be7240579d405c330845786a 2023-04-08 20:57:14,720 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/d5330807be7240579d405c330845786a, entries=150, sequenceid=111, filesize=11.6 K 2023-04-08 20:57:14,721 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/7bd446122bf546c89c0da304c2584963 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/7bd446122bf546c89c0da304c2584963 2023-04-08 20:57:14,723 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/7bd446122bf546c89c0da304c2584963, entries=150, sequenceid=111, filesize=11.6 K 2023-04-08 20:57:14,724 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/a915bdce531642a894e51bd7bb085b63 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/a915bdce531642a894e51bd7bb085b63 2023-04-08 20:57:14,728 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/a915bdce531642a894e51bd7bb085b63, entries=150, sequenceid=111, filesize=11.6 K 2023-04-08 20:57:14,728 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 0013c9607e8a284ca05e07f23e7866be in 1269ms, sequenceid=111, compaction requested=false 2023-04-08 20:57:14,728 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:14,728 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:14,728 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:14,729 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:14,729 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:14,729 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:14,729 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:14,729 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:14,729 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:14,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:14,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:14,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:14,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:14,730 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:14,730 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:14,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:14,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:14,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:14,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:14,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:14,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a8fa154[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:14,732 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:14,732 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:14,733 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:14,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:14,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:14,733 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:14,733 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:14,733 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:14,734 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:14,734 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:14,734 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:14,734 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:14,734 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:14,734 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:14,734 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:14,734 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:14,734 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:14,735 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:14,735 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:14,736 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:14,736 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:14,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:14,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:14,736 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:14,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:14,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:14,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:14,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:14,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:14,736 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:14,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:14,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:14,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:14,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:14,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:14,737 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:14,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:14,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:14,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:14,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:14,738 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:14,739 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:14,739 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:14,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:14,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:14,739 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:14,739 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:14,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:14,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:14,739 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:14,739 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:14,739 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:14,739 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:14,739 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:14,739 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:14,739 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:14,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:14,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:14,740 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:14,815 DEBUG [regionserver/jenkins-hbase20:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-08 20:57:14,823 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/03f61d02f4924fae8c9aa89ae7de19f2, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/54b42cda3aa7495faf6a5c98b5a78c37, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/239568da5d274a459d1ff4050c1558f1] to archive 2023-04-08 20:57:14,824 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 20:57:14,825 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/03f61d02f4924fae8c9aa89ae7de19f2 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/03f61d02f4924fae8c9aa89ae7de19f2 2023-04-08 20:57:14,827 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/54b42cda3aa7495faf6a5c98b5a78c37 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/54b42cda3aa7495faf6a5c98b5a78c37 2023-04-08 20:57:14,828 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/239568da5d274a459d1ff4050c1558f1 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/239568da5d274a459d1ff4050c1558f1 2023-04-08 20:57:14,835 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/e13e6aaa23064e3a8a3eb2c30c173178, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/1b2e9343bfa540c2ab3f40bede8050d5, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/36f31aa3328a4a08bd5531e56f1527e5] to archive 2023-04-08 20:57:14,836 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 20:57:14,837 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/e13e6aaa23064e3a8a3eb2c30c173178 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/e13e6aaa23064e3a8a3eb2c30c173178 2023-04-08 20:57:14,839 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/1b2e9343bfa540c2ab3f40bede8050d5 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/1b2e9343bfa540c2ab3f40bede8050d5 2023-04-08 20:57:14,839 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:14,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:14,841 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/36f31aa3328a4a08bd5531e56f1527e5 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/36f31aa3328a4a08bd5531e56f1527e5 2023-04-08 20:57:14,842 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:14,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:14,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:14,852 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/6009172b69ef4aca95bf19b50cd7ad7d, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/266288b8ad944bada79aca03fe1cea3f, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/676f1ca83f064f578320256d3b9202eb] to archive 2023-04-08 20:57:14,853 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:14,853 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:14,853 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:14,853 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:14,853 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:14,853 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:14,853 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 20:57:14,854 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:14,854 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:14,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:14,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:14,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:14,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:14,855 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/6009172b69ef4aca95bf19b50cd7ad7d to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/6009172b69ef4aca95bf19b50cd7ad7d 2023-04-08 20:57:14,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:14,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:14,855 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:14,855 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:14,855 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:14,855 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:14,855 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:14,855 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:14,855 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:14,855 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:14,855 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:14,855 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:14,855 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:14,855 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:14,856 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0013c9607e8a284ca05e07f23e7866be 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-08 20:57:14,856 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/266288b8ad944bada79aca03fe1cea3f to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/266288b8ad944bada79aca03fe1cea3f 2023-04-08 20:57:14,856 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=A 2023-04-08 20:57:14,856 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:14,856 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=B 2023-04-08 20:57:14,856 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:14,856 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=C 2023-04-08 20:57:14,856 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:14,857 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/676f1ca83f064f578320256d3b9202eb to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/676f1ca83f064f578320256d3b9202eb 2023-04-08 20:57:14,863 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/c9c359ce6ff94d6e9c28409d50e00739 2023-04-08 20:57:14,874 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/f55aae46c163470898316109daaed75a 2023-04-08 20:57:14,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 0013c9607e8a284ca05e07f23e7866be 2023-04-08 20:57:14,918 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:14,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680987434931, max timestamp=1680987434949Num uniques 859; 2023-04-08 20:57:14,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-08 20:57:14,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680987434931, max timestamp=1680987434949Num uniques 859; 2023-04-08 20:57:14,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-08 20:57:14,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680987434931, max timestamp=1680987434949Num uniques 859; 2023-04-08 20:57:14,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-08 20:57:14,979 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:14,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60798 deadline: 1680987494973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:14,979 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:14,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60824 deadline: 1680987494975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:14,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:14,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60788 deadline: 1680987494976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:15,084 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:15,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60798 deadline: 1680987495080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:15,085 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:15,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60824 deadline: 1680987495080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:15,085 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:15,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60788 deadline: 1680987495082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:15,283 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/a5ab98d7fd5b4c6e8568e6c830f74d52 2023-04-08 20:57:15,287 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/c9c359ce6ff94d6e9c28409d50e00739 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/c9c359ce6ff94d6e9c28409d50e00739 2023-04-08 20:57:15,288 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:15,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60824 deadline: 1680987495286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:15,288 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:15,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60788 deadline: 1680987495286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:15,290 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/c9c359ce6ff94d6e9c28409d50e00739, entries=150, sequenceid=123, filesize=11.6 K 2023-04-08 20:57:15,290 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/f55aae46c163470898316109daaed75a as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/f55aae46c163470898316109daaed75a 2023-04-08 20:57:15,291 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:15,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60798 deadline: 1680987495286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:15,293 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/f55aae46c163470898316109daaed75a, entries=150, sequenceid=123, filesize=11.6 K 2023-04-08 20:57:15,294 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/a5ab98d7fd5b4c6e8568e6c830f74d52 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/a5ab98d7fd5b4c6e8568e6c830f74d52 2023-04-08 20:57:15,296 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/a5ab98d7fd5b4c6e8568e6c830f74d52, entries=150, sequenceid=123, filesize=11.6 K 2023-04-08 20:57:15,297 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 0013c9607e8a284ca05e07f23e7866be in 442ms, sequenceid=123, compaction requested=true 2023-04-08 20:57:15,297 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:15,297 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:15,297 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:15,297 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:15,297 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:15,297 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:15,297 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:15,298 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:15,298 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:15,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:15,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:15,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:15,298 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:15,298 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:15,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:15,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:15,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:15,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:15,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:15,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:15,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@196f5a0c[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:15,299 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:15,299 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:15,300 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:15,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:15,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:15,300 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:15,300 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:15,300 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:15,300 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:15,300 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:15,300 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:15,300 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:15,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:15,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:15,301 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:15,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:15,301 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:15,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:15,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:15,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:15,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:15,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:15,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:15,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:15,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:15,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:15,303 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:15,303 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:15,303 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:15,303 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:15,303 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:15,303 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:15,304 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:15,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:15,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:15,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:15,304 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:15,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:15,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:15,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:15,304 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:15,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:15,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:15,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:15,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:15,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:15,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:15,305 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:15,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:15,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:15,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:15,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:15,306 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:15,306 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:15,306 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:15,306 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:15,312 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:15,312 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:15,312 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:15,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:15,312 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:15,312 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:15,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:15,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:15,312 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:15,312 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:15,312 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:15,312 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:15,312 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:15,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:15,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:15,413 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:15,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:15,415 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:15,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:15,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:15,416 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:15,416 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:15,417 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:15,417 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:15,417 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:15,417 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:15,418 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:15,418 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:15,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:15,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:15,419 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:15,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:15,419 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:15,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:15,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:15,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:15,419 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:15,419 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:15,419 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:15,419 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:15,419 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:15,420 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:15,420 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:15,420 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:15,420 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:15,420 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:15,420 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0013c9607e8a284ca05e07f23e7866be 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-08 20:57:15,420 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=A 2023-04-08 20:57:15,420 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:57:15,420 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=B 2023-04-08 20:57:15,420 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:57:15,420 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=C 2023-04-08 20:57:15,420 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:57:15,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 0013c9607e8a284ca05e07f23e7866be 2023-04-08 20:57:15,602 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:15,618 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:15,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60824 deadline: 1680987495611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:15,624 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:15,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60798 deadline: 1680987495618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:15,628 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:15,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60788 deadline: 1680987495618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:15,724 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:15,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60824 deadline: 1680987495719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:15,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:15,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60798 deadline: 1680987495725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:15,733 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:15,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60788 deadline: 1680987495729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:15,827 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/ccd459f4d4cd48c3b786612c668a0d4e 2023-04-08 20:57:15,929 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:15,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60824 deadline: 1680987495926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:15,937 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:15,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60798 deadline: 1680987495930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:15,938 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:15,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60788 deadline: 1680987495933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,235 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/fc240fb569894058b5a7280102df68a1 2023-04-08 20:57:16,236 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:16,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60824 deadline: 1680987496231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,244 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:16,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60798 deadline: 1680987496238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,246 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:16,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60788 deadline: 1680987496240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,644 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/3e199f3e6cde4e2f8cacb03766e5a6ed 2023-04-08 20:57:16,648 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/ccd459f4d4cd48c3b786612c668a0d4e as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/ccd459f4d4cd48c3b786612c668a0d4e 2023-04-08 20:57:16,652 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/ccd459f4d4cd48c3b786612c668a0d4e, entries=150, sequenceid=158, filesize=11.8 K 2023-04-08 20:57:16,653 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/fc240fb569894058b5a7280102df68a1 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/fc240fb569894058b5a7280102df68a1 2023-04-08 20:57:16,656 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/fc240fb569894058b5a7280102df68a1, entries=150, sequenceid=158, filesize=11.8 K 2023-04-08 20:57:16,657 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/3e199f3e6cde4e2f8cacb03766e5a6ed as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/3e199f3e6cde4e2f8cacb03766e5a6ed 2023-04-08 20:57:16,661 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/3e199f3e6cde4e2f8cacb03766e5a6ed, entries=150, sequenceid=158, filesize=11.8 K 2023-04-08 20:57:16,661 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for 0013c9607e8a284ca05e07f23e7866be in 1241ms, sequenceid=158, compaction requested=true 2023-04-08 20:57:16,661 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:16,661 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:16,661 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:16,661 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:16,662 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,662 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:16,662 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:16,663 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:16,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:16,663 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:16,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:16,663 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:16,663 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:16,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:16,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:16,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:16,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:16,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a5a97b9[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:16,664 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:16,665 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:16,665 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:16,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:16,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:16,665 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:16,665 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,665 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:16,665 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:16,665 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:16,666 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:16,666 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,666 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:16,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:16,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:16,666 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:16,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:16,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:16,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:16,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:16,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:16,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:16,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:16,668 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:16,668 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:16,668 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:16,668 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:16,668 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:16,669 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:16,669 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:16,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:16,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:16,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:16,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:16,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:16,669 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:16,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:16,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:16,669 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:16,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:16,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:16,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:16,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:16,671 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:16,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:16,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,672 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,672 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:16,672 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:16,672 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:16,672 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:16,672 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:16,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:16,672 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:16,672 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:16,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:16,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:16,672 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,673 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:16,673 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:16,673 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:16,673 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:16,673 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:16,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:16,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:16,747 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0013c9607e8a284ca05e07f23e7866be 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-08 20:57:16,747 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=A 2023-04-08 20:57:16,747 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:16,747 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=B 2023-04-08 20:57:16,747 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:16,747 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=C 2023-04-08 20:57:16,747 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:16,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 0013c9607e8a284ca05e07f23e7866be 2023-04-08 20:57:16,755 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/6d6776d152cb4a4798d78ae194ccb82c 2023-04-08 20:57:16,764 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/3d983b9d1b914ea8af998ebe0b19e045 2023-04-08 20:57:16,773 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:16,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:16,774 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:16,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:16,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:16,780 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:16,780 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:16,780 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:16,780 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:16,780 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:16,780 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:16,781 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,781 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:16,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:16,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:16,781 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,781 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:16,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:16,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:16,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:16,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:16,781 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:16,782 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:16,782 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:16,782 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:16,782 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:16,782 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:16,786 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:16,786 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:16,786 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:16,786 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:16,786 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:16,786 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:16,786 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:16,786 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:16,786 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,786 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:16,786 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:16,787 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,787 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:16,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:16,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:16,787 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:16,787 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:16,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:16,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:16,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:16,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:16,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:16,788 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:16,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2031d7a5[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:16,788 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:16,789 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:16,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:16,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:16,789 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:16,789 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:16,789 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,789 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:16,789 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:16,790 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,790 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:16,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:16,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:16,790 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:16,790 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:16,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:16,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:16,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:16,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:16,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:16,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:16,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:16,793 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:16,793 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:16,793 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:16,793 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:16,793 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:16,794 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:16,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:16,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:16,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:16,795 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:16,795 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:16,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:16,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:16,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:16,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:16,795 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:16,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:16,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:16,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:16,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:16,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,796 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:16,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:16,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,797 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,798 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:16,798 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:16,798 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,798 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:16,798 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:16,798 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:16,798 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:16,798 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:16,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:16,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:16,798 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:16,798 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:16,798 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:16,798 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:16,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:16,798 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:16,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:16,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680987436778, max timestamp=1680987436802Num uniques 859; 2023-04-08 20:57:16,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-08 20:57:16,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680987436778, max timestamp=1680987436802Num uniques 859; 2023-04-08 20:57:16,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-08 20:57:16,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680987436778, max timestamp=1680987436802Num uniques 859; 2023-04-08 20:57:16,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-08 20:57:16,841 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:16,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60824 deadline: 1680987496830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,841 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:16,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60788 deadline: 1680987496831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,843 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:16,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60798 deadline: 1680987496834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,898 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:16,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:16,900 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:16,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:16,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:16,901 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:16,901 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:16,901 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:16,901 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:16,902 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:16,902 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:16,902 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:16,902 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:16,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:16,903 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,903 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:16,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:16,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:16,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:16,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:16,903 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:16,903 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:16,903 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:16,903 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:16,904 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:16,904 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:16,904 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:16,904 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:16,904 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:16,904 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:16,904 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:16,904 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:16,904 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:16,904 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:16,904 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,904 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:16,904 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:16,905 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:16,905 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:16,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:16,905 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:16,905 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:16,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:16,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:16,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:16,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:16,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:16,906 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:16,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@372347f6[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:16,906 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:16,907 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:16,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:16,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:16,907 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:16,907 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:16,907 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:16,907 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,907 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:16,907 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,907 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:16,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:16,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:16,908 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:16,908 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:16,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:16,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:16,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:16,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:16,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:16,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:16,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:16,909 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:16,909 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:16,909 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:16,909 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:16,909 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:16,910 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:16,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:16,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:16,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:16,910 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:16,910 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:16,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:16,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:16,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:16,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:16,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:16,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:16,911 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:16,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:16,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:16,912 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:16,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:16,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,913 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:16,913 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:16,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:16,913 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:16,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:16,913 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:16,913 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:16,914 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:16,914 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:16,914 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:16,914 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,914 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:16,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:16,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:16,914 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:16,914 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:16,914 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:16,914 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:16,947 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:16,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60788 deadline: 1680987496942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,948 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:16,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60824 deadline: 1680987496942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:16,948 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:16,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60798 deadline: 1680987496944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,014 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:17,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:17,016 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:17,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:17,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:17,017 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:17,017 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:17,017 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:17,017 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:17,017 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:17,017 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:17,018 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:17,018 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:17,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:17,018 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,018 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:17,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:17,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:17,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:17,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:17,019 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:17,019 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:17,019 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:17,019 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:17,019 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:17,019 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:17,019 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:17,019 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:17,019 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:17,019 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:17,019 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:17,019 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:17,019 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:17,019 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:17,019 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,019 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:17,019 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:17,021 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,021 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:17,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:17,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:17,021 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:17,021 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:17,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:17,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:17,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:17,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:17,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:17,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54ffc424[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:17,023 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:17,023 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:17,024 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,024 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:17,025 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:17,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:17,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:17,025 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:17,025 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:17,025 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:17,025 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,025 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:17,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:17,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:17,025 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:17,026 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:17,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:17,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:17,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:17,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:17,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:17,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:17,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:17,027 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:17,027 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:17,027 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:17,027 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:17,027 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:17,028 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:17,028 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:17,028 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:17,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:17,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:17,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:17,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:17,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:17,029 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:17,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:17,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:17,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:17,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:17,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:17,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:17,029 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:17,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:17,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,031 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,031 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:17,031 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:17,031 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,031 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:17,031 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:17,031 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:17,031 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:17,031 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:17,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:17,031 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:17,031 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:17,031 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:17,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:17,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:17,031 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:17,031 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:17,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:17,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:17,132 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:17,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:17,133 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:17,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:17,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:17,135 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:17,135 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:17,135 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:17,135 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:17,135 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:17,135 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:17,136 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:17,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:17,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:17,136 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:17,136 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,137 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:17,137 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:17,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:17,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:17,137 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:17,137 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:17,137 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:17,137 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:17,137 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:17,138 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:17,138 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:17,138 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:17,138 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:17,138 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:17,138 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:17,138 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:17,138 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:17,138 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:17,138 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,138 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:17,138 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:17,139 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,139 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:17,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:17,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:17,139 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:17,139 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:17,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:17,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:17,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:17,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:17,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:17,140 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:17,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a86765d[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:17,140 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:17,141 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:17,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:17,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:17,141 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:17,141 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:17,141 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,141 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:17,141 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:17,142 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,142 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:17,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:17,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:17,142 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:17,142 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:17,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:17,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:17,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:17,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:17,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:17,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:17,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:17,144 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:17,144 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:17,144 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:17,144 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:17,144 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:17,145 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:17,145 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:17,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:17,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:17,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:17,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:17,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:17,145 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:17,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:17,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:17,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:17,146 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:17,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:17,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:17,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:17,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,147 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:17,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:17,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,148 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:17,148 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:17,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:17,148 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:17,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:17,148 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:17,148 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:17,148 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:17,148 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:17,148 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:17,148 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,148 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:17,148 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:17,148 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:17,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:17,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:17,148 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:17,149 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:17,153 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:17,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60788 deadline: 1680987497149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,153 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:17,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60824 deadline: 1680987497149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,153 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:17,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60798 deadline: 1680987497150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,173 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/fb0c90f524f541d787a6a7e8bf126d13 2023-04-08 20:57:17,175 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/6d6776d152cb4a4798d78ae194ccb82c as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/6d6776d152cb4a4798d78ae194ccb82c 2023-04-08 20:57:17,178 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/6d6776d152cb4a4798d78ae194ccb82c, entries=200, sequenceid=169, filesize=14.1 K 2023-04-08 20:57:17,179 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/3d983b9d1b914ea8af998ebe0b19e045 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/3d983b9d1b914ea8af998ebe0b19e045 2023-04-08 20:57:17,181 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/3d983b9d1b914ea8af998ebe0b19e045, entries=150, sequenceid=169, filesize=11.8 K 2023-04-08 20:57:17,181 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/fb0c90f524f541d787a6a7e8bf126d13 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/fb0c90f524f541d787a6a7e8bf126d13 2023-04-08 20:57:17,184 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/fb0c90f524f541d787a6a7e8bf126d13, entries=150, sequenceid=169, filesize=11.8 K 2023-04-08 20:57:17,184 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 0013c9607e8a284ca05e07f23e7866be in 437ms, sequenceid=169, compaction requested=true 2023-04-08 20:57:17,184 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:17,185 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:57:17,185 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:57:17,185 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-08 20:57:17,185 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:57:17,185 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-08 20:57:17,186 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62223 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-08 20:57:17,186 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59833 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-08 20:57:17,186 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): 0013c9607e8a284ca05e07f23e7866be/A is initiating minor compaction (all files) 2023-04-08 20:57:17,186 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 0013c9607e8a284ca05e07f23e7866be/B is initiating minor compaction (all files) 2023-04-08 20:57:17,186 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0013c9607e8a284ca05e07f23e7866be/A in TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:17,186 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0013c9607e8a284ca05e07f23e7866be/B in TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:17,186 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/8e508f63691e4e558db0f7a83d2f832a, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/d5330807be7240579d405c330845786a, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/c9c359ce6ff94d6e9c28409d50e00739, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/ccd459f4d4cd48c3b786612c668a0d4e, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/6d6776d152cb4a4798d78ae194ccb82c] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp, totalSize=60.8 K 2023-04-08 20:57:17,186 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/58e8e8f8b93a4a3292a08281b3fdd82a, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/7bd446122bf546c89c0da304c2584963, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/f55aae46c163470898316109daaed75a, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/fc240fb569894058b5a7280102df68a1, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/3d983b9d1b914ea8af998ebe0b19e045] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp, totalSize=58.4 K 2023-04-08 20:57:17,186 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 8e508f63691e4e558db0f7a83d2f832a, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=74, earliestPutTs=1680987432544 2023-04-08 20:57:17,186 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 58e8e8f8b93a4a3292a08281b3fdd82a, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=74, earliestPutTs=1680987432544 2023-04-08 20:57:17,186 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting d5330807be7240579d405c330845786a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1680987432635 2023-04-08 20:57:17,186 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 7bd446122bf546c89c0da304c2584963, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1680987432635 2023-04-08 20:57:17,187 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting c9c359ce6ff94d6e9c28409d50e00739, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1680987433786 2023-04-08 20:57:17,187 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting f55aae46c163470898316109daaed75a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1680987433786 2023-04-08 20:57:17,187 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting ccd459f4d4cd48c3b786612c668a0d4e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1680987434971 2023-04-08 20:57:17,187 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting fc240fb569894058b5a7280102df68a1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1680987434971 2023-04-08 20:57:17,187 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 6d6776d152cb4a4798d78ae194ccb82c, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1680987435606 2023-04-08 20:57:17,187 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 3d983b9d1b914ea8af998ebe0b19e045, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1680987435610 2023-04-08 20:57:17,193 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0013c9607e8a284ca05e07f23e7866be#A#compaction#376 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:57:17,194 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 0013c9607e8a284ca05e07f23e7866be#B#compaction#377 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:57:17,249 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:17,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:17,251 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:17,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:17,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:17,252 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:17,252 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:17,252 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:17,252 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:17,252 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:17,252 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:17,253 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:17,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:17,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:17,253 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:17,253 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,253 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:17,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:17,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:17,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:17,254 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:17,254 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:17,254 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:17,254 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:17,254 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:17,254 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:17,254 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:17,254 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:17,254 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:17,254 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:17,254 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0013c9607e8a284ca05e07f23e7866be 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-08 20:57:17,254 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=A 2023-04-08 20:57:17,255 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:57:17,255 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=B 2023-04-08 20:57:17,255 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:57:17,255 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=C 2023-04-08 20:57:17,255 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:57:17,260 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/e6405254d9a042cb9e572a4235d7dbce 2023-04-08 20:57:17,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 0013c9607e8a284ca05e07f23e7866be 2023-04-08 20:57:17,458 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:17,477 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:17,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60788 deadline: 1680987497469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:17,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60824 deadline: 1680987497472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,485 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:17,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60798 deadline: 1680987497478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:17,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60788 deadline: 1680987497578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:17,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60824 deadline: 1680987497579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,594 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:17,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60798 deadline: 1680987497586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,603 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/fd074d2da18640f4b92b22a6612af1f4 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/fd074d2da18640f4b92b22a6612af1f4 2023-04-08 20:57:17,604 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/4f092bb0d2d24a07929c0cddd75cce51 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/4f092bb0d2d24a07929c0cddd75cce51 2023-04-08 20:57:17,611 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 0013c9607e8a284ca05e07f23e7866be/B of 0013c9607e8a284ca05e07f23e7866be into fd074d2da18640f4b92b22a6612af1f4(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:57:17,612 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:17,612 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be., storeName=0013c9607e8a284ca05e07f23e7866be/B, priority=11, startTime=1680987437185; duration=0sec 2023-04-08 20:57:17,612 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:57:17,612 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-08 20:57:17,614 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59833 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-08 20:57:17,614 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 0013c9607e8a284ca05e07f23e7866be/C is initiating minor compaction (all files) 2023-04-08 20:57:17,614 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0013c9607e8a284ca05e07f23e7866be/C in TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:17,614 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/8a9d563c538941238460b51ebfafed8f, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/a915bdce531642a894e51bd7bb085b63, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/a5ab98d7fd5b4c6e8568e6c830f74d52, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/3e199f3e6cde4e2f8cacb03766e5a6ed, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/fb0c90f524f541d787a6a7e8bf126d13] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp, totalSize=58.4 K 2023-04-08 20:57:17,614 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 8a9d563c538941238460b51ebfafed8f, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=74, earliestPutTs=1680987432544 2023-04-08 20:57:17,615 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting a915bdce531642a894e51bd7bb085b63, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1680987432635 2023-04-08 20:57:17,615 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 0013c9607e8a284ca05e07f23e7866be/A of 0013c9607e8a284ca05e07f23e7866be into 4f092bb0d2d24a07929c0cddd75cce51(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:57:17,615 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:17,615 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting a5ab98d7fd5b4c6e8568e6c830f74d52, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1680987433786 2023-04-08 20:57:17,615 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be., storeName=0013c9607e8a284ca05e07f23e7866be/A, priority=11, startTime=1680987437185; duration=0sec 2023-04-08 20:57:17,615 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:57:17,615 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 3e199f3e6cde4e2f8cacb03766e5a6ed, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1680987434971 2023-04-08 20:57:17,615 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting fb0c90f524f541d787a6a7e8bf126d13, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1680987435610 2023-04-08 20:57:17,626 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 0013c9607e8a284ca05e07f23e7866be#C#compaction#380 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:57:17,669 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/7eef716d7df642d3837163baa032ef7e 2023-04-08 20:57:17,791 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:17,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60824 deadline: 1680987497786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,791 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:17,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60788 deadline: 1680987497787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:17,798 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:17,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60798 deadline: 1680987497796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,034 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/a0a252e001c54dc897ba0db5ae487417 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/a0a252e001c54dc897ba0db5ae487417 2023-04-08 20:57:18,037 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 0013c9607e8a284ca05e07f23e7866be/C of 0013c9607e8a284ca05e07f23e7866be into a0a252e001c54dc897ba0db5ae487417(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:57:18,037 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:18,037 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be., storeName=0013c9607e8a284ca05e07f23e7866be/C, priority=11, startTime=1680987437185; duration=0sec 2023-04-08 20:57:18,037 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:57:18,077 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/c71a49945aae4a158380a5e26d1e8365 2023-04-08 20:57:18,081 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/e6405254d9a042cb9e572a4235d7dbce as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/e6405254d9a042cb9e572a4235d7dbce 2023-04-08 20:57:18,083 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/e6405254d9a042cb9e572a4235d7dbce, entries=150, sequenceid=203, filesize=11.8 K 2023-04-08 20:57:18,084 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/7eef716d7df642d3837163baa032ef7e as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/7eef716d7df642d3837163baa032ef7e 2023-04-08 20:57:18,087 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/7eef716d7df642d3837163baa032ef7e, entries=150, sequenceid=203, filesize=11.8 K 2023-04-08 20:57:18,087 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/c71a49945aae4a158380a5e26d1e8365 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/c71a49945aae4a158380a5e26d1e8365 2023-04-08 20:57:18,089 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/c71a49945aae4a158380a5e26d1e8365, entries=150, sequenceid=203, filesize=11.8 K 2023-04-08 20:57:18,090 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for 0013c9607e8a284ca05e07f23e7866be in 836ms, sequenceid=203, compaction requested=false 2023-04-08 20:57:18,090 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:18,090 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:18,090 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:18,090 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:18,090 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,091 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:18,091 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:18,091 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,091 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:18,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:18,092 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,092 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:18,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:18,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:18,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:18,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:18,093 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:18,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64241c57[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:18,093 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,093 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,093 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:18,093 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:18,093 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:18,094 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,094 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:18,094 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,094 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:18,094 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:18,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:18,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:18,094 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:18,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:18,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:18,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:18,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 0013c9607e8a284ca05e07f23e7866be 2023-04-08 20:57:18,096 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0013c9607e8a284ca05e07f23e7866be 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-08 20:57:18,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,096 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=A 2023-04-08 20:57:18,096 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:18,096 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=B 2023-04-08 20:57:18,096 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:18,096 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=C 2023-04-08 20:57:18,096 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:18,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:18,096 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:18,096 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:18,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:18,096 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:18,097 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:18,097 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:18,098 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,098 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,098 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:18,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:18,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:18,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:18,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:18,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:18,099 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:18,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,101 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:18,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,104 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/5a838769bf93483da8947c8f75fd342c 2023-04-08 20:57:18,105 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,105 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:18,105 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:18,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:18,105 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:18,105 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:18,105 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:18,106 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:18,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:18,105 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:18,106 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,106 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,106 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:18,106 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:18,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:18,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:18,106 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,106 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:18,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60798 deadline: 1680987498132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:18,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60788 deadline: 1680987498133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,144 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:18,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60824 deadline: 1680987498137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,206 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:18,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:18,208 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:18,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:18,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:18,208 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:18,209 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:18,209 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:18,209 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:18,209 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,209 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:18,210 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,210 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:18,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:18,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:18,210 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,210 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:18,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:18,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:18,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:18,210 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:18,211 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:18,211 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:18,211 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:18,211 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:18,211 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:18,211 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:18,211 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:18,211 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:18,211 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:18,211 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:18,211 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:18,211 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:18,211 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:18,211 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,211 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:18,211 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:18,212 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:18,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:18,212 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:18,212 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,212 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:18,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:18,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:18,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:18,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4915ebd2[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:18,213 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:18,213 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,213 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,213 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:18,213 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,213 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:18,213 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:18,214 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:18,215 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,215 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:18,215 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:18,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:18,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:18,215 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:18,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:18,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:18,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:18,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:18,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:18,216 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:18,216 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:18,216 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:18,216 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:18,216 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:18,217 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,217 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:18,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:18,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,217 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:18,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:18,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:18,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:18,217 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:18,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,218 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:18,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,219 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,220 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:18,220 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:18,220 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:18,220 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:18,220 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:18,220 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:18,220 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:18,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:18,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:18,220 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,220 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,220 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,220 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,220 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:18,220 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:18,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:18,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:18,243 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:18,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60798 deadline: 1680987498240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:18,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60788 deadline: 1680987498240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,248 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:18,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60824 deadline: 1680987498245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,320 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:18,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:18,322 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:18,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:18,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:18,322 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:18,323 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:18,323 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:18,323 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:18,323 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,323 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:18,324 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,324 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:18,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:18,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:18,324 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,324 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:18,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:18,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:18,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:18,325 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:18,325 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:18,325 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:18,325 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:18,325 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:18,325 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:18,325 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:18,325 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:18,325 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:18,325 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:18,325 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:18,325 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:18,325 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:18,325 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:18,325 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,325 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:18,325 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:18,326 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,326 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:18,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:18,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:18,326 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,327 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:18,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:18,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:18,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:18,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6361c830[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:18,328 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:18,328 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,328 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,328 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:18,328 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:18,328 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,328 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:18,328 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:18,329 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:18,329 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:18,329 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:18,329 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:18,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:18,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:18,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:18,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:18,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:18,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:18,331 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:18,331 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:18,331 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:18,331 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:18,331 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:18,332 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,332 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,332 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:18,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:18,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:18,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:18,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:18,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:18,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,332 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:18,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,333 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:18,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,334 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:18,334 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,334 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:18,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:18,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:18,335 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:18,335 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:18,335 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:18,335 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:18,335 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:18,335 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,335 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:18,335 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,335 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:18,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:18,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:18,335 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,335 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,435 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:18,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:18,437 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:18,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:18,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:18,437 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:18,438 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:18,438 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:18,438 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:18,438 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,438 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:18,439 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:18,439 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:18,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:18,439 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,439 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:18,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:18,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:18,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:18,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:18,440 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:18,440 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:18,440 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:18,440 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:18,440 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:18,440 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:18,440 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:18,440 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:18,440 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:18,440 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:18,440 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:18,440 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:18,440 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,440 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:18,440 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:18,441 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,441 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:18,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:18,441 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,441 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:18,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:18,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:18,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:18,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:18,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1aac6e9f[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:18,442 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:18,442 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,443 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,443 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:18,443 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:18,443 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:18,443 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,443 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:18,444 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,444 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:18,444 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:18,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:18,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:18,444 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:18,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:18,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:18,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:18,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:18,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:18,446 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:18,446 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:18,446 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:18,446 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:18,446 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:18,446 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:18,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60798 deadline: 1680987498444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,447 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:18,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:18,447 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,447 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:18,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:18,447 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:18,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:18,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:18,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,448 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:18,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,450 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:18,450 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60788 deadline: 1680987498445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,450 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:18,450 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:18,450 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,450 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,450 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,450 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,450 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:18,450 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:18,450 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:18,451 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:18,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:18,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:18,451 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:18,451 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:18,451 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:18,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:18,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:18,453 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:18,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60824 deadline: 1680987498449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,477 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:18,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60764 deadline: 1680987498476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,479 DEBUG [Thread-2751] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8156 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be., hostname=jenkins-hbase20.apache.org,41091,1680987313601, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 20:57:18,498 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:18,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60828 deadline: 1680987498497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,500 DEBUG [Thread-2745] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8178 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be., hostname=jenkins-hbase20.apache.org,41091,1680987313601, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 20:57:18,514 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/a5aeb67cd58d44829103d32d7d99153f 2023-04-08 20:57:18,551 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:18,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:18,553 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:18,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:18,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:18,554 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:18,554 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:18,554 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:18,554 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:18,554 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,554 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:18,555 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,555 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:18,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:18,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:18,555 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,555 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:18,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:18,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:18,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:18,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:18,556 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:18,556 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:18,556 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:18,556 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:18,556 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:18,556 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:18,556 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:18,556 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:18,556 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:18,556 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:18,556 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:18,556 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:18,556 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:18,556 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,556 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:18,556 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:18,557 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,557 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:18,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:18,557 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,557 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:18,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:18,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:18,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:18,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5eae1f39[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:18,558 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:18,558 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,559 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,559 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:18,559 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,559 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:18,559 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:18,559 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:18,559 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:18,560 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:18,559 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,560 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:18,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:18,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:18,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:18,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:18,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:18,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:18,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:18,561 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:18,561 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:18,561 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:18,561 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:18,561 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:18,562 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,563 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:18,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:18,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,563 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:18,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:18,563 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:18,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:18,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:18,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,564 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:18,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,565 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,565 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:18,565 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:18,565 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,565 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,565 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,565 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:18,565 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:18,565 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:18,565 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,565 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:18,566 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:18,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:18,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:18,566 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:18,566 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:18,566 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:18,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:18,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:18,666 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:18,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:18,668 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:18,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:18,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:18,668 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:18,669 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:18,669 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:18,669 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:18,669 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,669 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:18,670 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,670 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:18,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:18,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:18,670 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,670 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:18,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:18,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:18,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:18,671 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:18,671 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:18,671 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:18,671 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:18,671 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:18,671 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:18,671 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:18,671 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:18,671 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:18,671 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:18,671 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:18,671 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:18,671 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:18,671 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:18,671 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,671 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:18,671 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:18,672 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,672 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:18,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:18,672 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,672 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:18,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:18,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:18,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:18,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:18,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38832445[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:18,673 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:18,673 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,674 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,674 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:18,674 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:18,674 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:18,674 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,674 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:18,675 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,675 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:18,675 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:18,675 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:18,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:18,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:18,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:18,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:18,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:18,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:18,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:18,678 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:18,678 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:18,678 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:18,678 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:18,678 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:18,679 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:18,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:18,679 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,679 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:18,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:18,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:18,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:18,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:18,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,686 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:18,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,688 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,689 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:18,689 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:18,689 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:18,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:18,689 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:18,690 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:18,689 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:18,689 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:18,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:18,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:18,690 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:18,690 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:18,690 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:18,690 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,690 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,690 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,690 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:18,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60798 deadline: 1680987498748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:18,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60788 deadline: 1680987498751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,755 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:18,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60824 deadline: 1680987498753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,790 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:18,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:18,792 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:18,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:18,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:18,793 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:18,793 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:18,793 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:18,793 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:18,793 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,793 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:18,794 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:18,794 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:18,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:18,794 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,794 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:18,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:18,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:18,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:18,795 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:18,795 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:18,795 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:18,795 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:18,795 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:18,795 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:18,795 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:18,795 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:18,795 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:18,795 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:18,795 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:18,795 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:18,795 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:18,795 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:18,796 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,796 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:18,796 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:18,796 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,796 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:18,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:18,797 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,797 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:18,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:18,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:18,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:18,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:18,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55ce4b2d[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:18,798 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:18,798 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,798 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,798 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:18,799 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,799 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:18,799 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:18,799 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:18,799 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:18,799 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,799 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:18,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:18,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:18,799 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:18,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:18,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:18,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:18,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:18,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:18,802 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:18,802 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:18,802 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:18,802 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:18,802 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:18,803 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,803 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:18,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:18,803 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:18,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:18,803 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:18,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:18,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:18,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,804 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:18,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,805 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,805 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:18,805 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:18,805 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:18,805 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:18,805 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:18,805 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:18,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:18,806 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:18,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:18,806 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,806 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:18,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:18,806 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,806 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:18,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:18,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:18,806 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,806 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,906 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:18,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:18,908 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:18,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:18,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:18,909 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:18,909 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:18,909 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:18,909 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:18,909 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,909 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:18,910 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:18,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:18,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:18,910 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:18,910 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,910 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:18,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:18,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:18,911 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:18,911 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:18,911 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:18,911 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:18,911 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:18,911 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:18,911 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:18,911 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:18,911 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:18,911 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:18,911 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:18,911 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:18,911 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:18,911 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:18,912 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,912 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:18,912 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:18,912 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,912 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:18,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:18,913 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:18,913 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:18,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:18,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:18,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:18,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cef0f9d[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:18,913 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:18,914 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,914 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,914 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:18,914 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:18,914 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:18,914 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,914 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:18,915 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,915 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:18,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:18,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:18,915 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:18,915 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:18,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:18,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:18,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:18,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:18,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:18,917 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:18,917 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:18,917 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:18,917 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:18,917 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:18,917 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:18,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:18,917 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,918 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:18,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:18,918 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:18,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:18,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:18,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,919 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:18,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:18,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,920 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,920 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:18,920 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:18,920 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:18,920 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,920 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:18,920 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:18,920 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:18,921 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:18,920 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:18,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:18,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:18,921 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:18,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:18,921 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:18,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:18,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:18,921 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:18,921 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:18,923 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/ba91a4d6099f4fb78196dbabe70bd690 2023-04-08 20:57:18,926 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/5a838769bf93483da8947c8f75fd342c as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/5a838769bf93483da8947c8f75fd342c 2023-04-08 20:57:18,929 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/5a838769bf93483da8947c8f75fd342c, entries=200, sequenceid=218, filesize=14.1 K 2023-04-08 20:57:18,930 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/a5aeb67cd58d44829103d32d7d99153f as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/a5aeb67cd58d44829103d32d7d99153f 2023-04-08 20:57:18,942 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/a5aeb67cd58d44829103d32d7d99153f, entries=150, sequenceid=218, filesize=11.8 K 2023-04-08 20:57:18,943 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/ba91a4d6099f4fb78196dbabe70bd690 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/ba91a4d6099f4fb78196dbabe70bd690 2023-04-08 20:57:18,947 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/ba91a4d6099f4fb78196dbabe70bd690, entries=150, sequenceid=218, filesize=11.8 K 2023-04-08 20:57:18,948 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 0013c9607e8a284ca05e07f23e7866be in 852ms, sequenceid=218, compaction requested=true 2023-04-08 20:57:18,948 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:18,948 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:57:18,948 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:57:18,948 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:57:18,948 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:57:18,948 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:57:18,953 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38670 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:57:18,953 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): 0013c9607e8a284ca05e07f23e7866be/A is initiating minor compaction (all files) 2023-04-08 20:57:18,953 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36280 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:57:18,953 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0013c9607e8a284ca05e07f23e7866be/A in TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:18,953 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 0013c9607e8a284ca05e07f23e7866be/B is initiating minor compaction (all files) 2023-04-08 20:57:18,953 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0013c9607e8a284ca05e07f23e7866be/B in TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:18,953 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/4f092bb0d2d24a07929c0cddd75cce51, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/e6405254d9a042cb9e572a4235d7dbce, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/5a838769bf93483da8947c8f75fd342c] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp, totalSize=37.8 K 2023-04-08 20:57:18,953 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/fd074d2da18640f4b92b22a6612af1f4, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/7eef716d7df642d3837163baa032ef7e, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/a5aeb67cd58d44829103d32d7d99153f] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp, totalSize=35.4 K 2023-04-08 20:57:18,953 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 4f092bb0d2d24a07929c0cddd75cce51, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1680987435610 2023-04-08 20:57:18,953 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting fd074d2da18640f4b92b22a6612af1f4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1680987435610 2023-04-08 20:57:18,954 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting e6405254d9a042cb9e572a4235d7dbce, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680987436829 2023-04-08 20:57:18,954 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 7eef716d7df642d3837163baa032ef7e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680987436829 2023-04-08 20:57:18,954 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 5a838769bf93483da8947c8f75fd342c, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680987437467 2023-04-08 20:57:18,954 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting a5aeb67cd58d44829103d32d7d99153f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680987437471 2023-04-08 20:57:18,959 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 0013c9607e8a284ca05e07f23e7866be#B#compaction#385 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:57:18,960 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0013c9607e8a284ca05e07f23e7866be#A#compaction#386 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:57:19,021 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:19,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:19,023 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:19,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:19,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:19,024 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:19,024 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:19,024 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:19,024 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:19,024 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:19,024 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:19,025 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:19,025 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:19,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:19,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:19,025 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:19,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:19,025 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:19,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:19,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:19,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:19,026 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:19,026 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:19,026 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:19,026 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:19,026 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:19,026 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:19,026 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:19,027 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:19,027 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:19,027 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:19,027 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0013c9607e8a284ca05e07f23e7866be 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-08 20:57:19,027 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=A 2023-04-08 20:57:19,027 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:19,027 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=B 2023-04-08 20:57:19,027 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:19,027 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=C 2023-04-08 20:57:19,027 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:19,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 0013c9607e8a284ca05e07f23e7866be 2023-04-08 20:57:19,260 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:19,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:19,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987499269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:19,275 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:19,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60788 deadline: 1680987499272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:19,278 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:19,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987499272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:19,366 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/649fbd0af4e447fea3389ca1419c22e9 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/649fbd0af4e447fea3389ca1419c22e9 2023-04-08 20:57:19,369 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0013c9607e8a284ca05e07f23e7866be/B of 0013c9607e8a284ca05e07f23e7866be into 649fbd0af4e447fea3389ca1419c22e9(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:57:19,369 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:19,369 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be., storeName=0013c9607e8a284ca05e07f23e7866be/B, priority=13, startTime=1680987438948; duration=0sec 2023-04-08 20:57:19,369 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:57:19,369 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:57:19,370 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36280 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:57:19,370 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 0013c9607e8a284ca05e07f23e7866be/C is initiating minor compaction (all files) 2023-04-08 20:57:19,371 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0013c9607e8a284ca05e07f23e7866be/C in TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:19,371 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/a0a252e001c54dc897ba0db5ae487417, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/c71a49945aae4a158380a5e26d1e8365, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/ba91a4d6099f4fb78196dbabe70bd690] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp, totalSize=35.4 K 2023-04-08 20:57:19,371 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting a0a252e001c54dc897ba0db5ae487417, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1680987435610 2023-04-08 20:57:19,371 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting c71a49945aae4a158380a5e26d1e8365, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680987436829 2023-04-08 20:57:19,372 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting ba91a4d6099f4fb78196dbabe70bd690, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680987437471 2023-04-08 20:57:19,374 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/ec5c3899e7d745f4b2aa5fd197533ee1 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/ec5c3899e7d745f4b2aa5fd197533ee1 2023-04-08 20:57:19,376 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 0013c9607e8a284ca05e07f23e7866be#C#compaction#388 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:57:19,377 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:19,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987499373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:19,377 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:19,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987499376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:19,378 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0013c9607e8a284ca05e07f23e7866be/A of 0013c9607e8a284ca05e07f23e7866be into ec5c3899e7d745f4b2aa5fd197533ee1(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:57:19,378 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:19,378 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be., storeName=0013c9607e8a284ca05e07f23e7866be/A, priority=13, startTime=1680987438948; duration=0sec 2023-04-08 20:57:19,378 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:57:19,384 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:19,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987499379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:19,443 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/321999a731bb494fa12c3c5d1f8d8db8 2023-04-08 20:57:19,581 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:19,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987499578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:19,581 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:19,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987499578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:19,586 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:19,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987499585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:19,784 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/90934e93d68743d4b72b4cdeb7046d35 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/90934e93d68743d4b72b4cdeb7046d35 2023-04-08 20:57:19,787 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0013c9607e8a284ca05e07f23e7866be/C of 0013c9607e8a284ca05e07f23e7866be into 90934e93d68743d4b72b4cdeb7046d35(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:57:19,788 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:19,788 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be., storeName=0013c9607e8a284ca05e07f23e7866be/C, priority=13, startTime=1680987438948; duration=0sec 2023-04-08 20:57:19,788 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:57:19,851 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/22617e11ffac47bf977e0533d1a35d39 2023-04-08 20:57:19,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:19,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987499883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:19,887 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:19,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987499884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:19,892 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:19,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987499886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:20,259 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/ddfd72e78aff4b7abe3ae11634a808bc 2023-04-08 20:57:20,262 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/321999a731bb494fa12c3c5d1f8d8db8 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/321999a731bb494fa12c3c5d1f8d8db8 2023-04-08 20:57:20,265 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/321999a731bb494fa12c3c5d1f8d8db8, entries=150, sequenceid=238, filesize=11.8 K 2023-04-08 20:57:20,265 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/22617e11ffac47bf977e0533d1a35d39 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/22617e11ffac47bf977e0533d1a35d39 2023-04-08 20:57:20,268 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/22617e11ffac47bf977e0533d1a35d39, entries=150, sequenceid=238, filesize=11.8 K 2023-04-08 20:57:20,269 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/ddfd72e78aff4b7abe3ae11634a808bc as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/ddfd72e78aff4b7abe3ae11634a808bc 2023-04-08 20:57:20,272 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/ddfd72e78aff4b7abe3ae11634a808bc, entries=150, sequenceid=238, filesize=11.8 K 2023-04-08 20:57:20,272 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 0013c9607e8a284ca05e07f23e7866be in 1245ms, sequenceid=238, compaction requested=false 2023-04-08 20:57:20,272 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:20,272 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:20,272 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:20,272 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:20,272 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:20,273 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:20,273 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:20,274 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:20,274 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:20,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:20,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:20,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:20,274 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:20,274 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:20,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:20,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:20,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:20,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:20,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:20,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:20,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2969aee[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:20,275 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:20,275 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:20,276 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:20,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:20,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:20,276 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:20,276 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:20,276 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:20,276 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:20,276 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:20,276 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:20,277 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:20,277 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:20,277 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:20,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:20,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:20,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:20,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:20,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:20,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:20,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:20,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:20,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:20,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:20,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:20,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:20,279 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:20,279 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:20,279 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:20,279 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:20,279 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:20,280 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:20,280 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:20,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:20,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:20,280 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:20,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:20,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:20,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:20,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:20,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:20,281 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:20,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:20,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:20,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:20,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:20,281 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:20,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:20,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:20,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:20,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:20,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:20,283 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:20,283 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:20,283 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:20,283 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:20,283 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:20,283 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:20,283 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:20,283 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:20,283 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:20,283 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:20,283 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:20,283 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:20,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:20,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:20,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:20,283 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:20,283 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:20,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:20,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:20,383 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:20,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:20,385 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:20,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:20,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:20,386 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:20,386 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:20,386 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:20,386 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:20,386 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:20,386 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:20,387 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:20,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:20,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:20,387 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:20,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:20,387 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:20,387 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:20,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:20,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:20,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:20,388 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:20,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:20,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:20,388 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:20,388 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:20,388 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:20,389 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:20,389 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:20,389 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:20,389 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:20,389 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0013c9607e8a284ca05e07f23e7866be 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-08 20:57:20,389 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=A 2023-04-08 20:57:20,389 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:20,389 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=B 2023-04-08 20:57:20,389 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:20,389 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=C 2023-04-08 20:57:20,389 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:20,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 0013c9607e8a284ca05e07f23e7866be 2023-04-08 20:57:20,392 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:20,397 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/1b02e15a972e44fc8b0d3d8314956966 2023-04-08 20:57:20,418 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:20,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987500414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:20,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:20,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987500415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:20,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:20,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987500416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:20,522 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:20,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987500519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:20,523 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:20,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987500521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:20,523 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:20,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987500521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:20,728 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:20,728 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:20,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987500724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:20,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987500725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:20,728 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:20,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987500725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:20,806 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/496ac6ea10624e4c81265a6a0b80c79a 2023-04-08 20:57:21,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:21,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987501030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:21,033 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:21,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987501031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:21,033 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:21,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987501031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:21,214 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/ad905d732fcd494981238323edb53537 2023-04-08 20:57:21,217 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/1b02e15a972e44fc8b0d3d8314956966 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/1b02e15a972e44fc8b0d3d8314956966 2023-04-08 20:57:21,220 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/1b02e15a972e44fc8b0d3d8314956966, entries=150, sequenceid=253, filesize=11.8 K 2023-04-08 20:57:21,221 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/496ac6ea10624e4c81265a6a0b80c79a as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/496ac6ea10624e4c81265a6a0b80c79a 2023-04-08 20:57:21,224 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/496ac6ea10624e4c81265a6a0b80c79a, entries=150, sequenceid=253, filesize=11.8 K 2023-04-08 20:57:21,224 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/ad905d732fcd494981238323edb53537 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/ad905d732fcd494981238323edb53537 2023-04-08 20:57:21,228 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/ad905d732fcd494981238323edb53537, entries=150, sequenceid=253, filesize=11.8 K 2023-04-08 20:57:21,228 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 0013c9607e8a284ca05e07f23e7866be in 839ms, sequenceid=253, compaction requested=true 2023-04-08 20:57:21,229 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:21,229 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:21,229 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:21,229 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:21,229 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:21,229 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:21,229 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:21,230 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:21,230 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:21,230 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:21,230 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:21,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:21,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:21,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:21,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:21,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:21,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:21,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:21,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:21,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:21,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@231627f5[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:21,232 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:21,232 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:21,232 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:21,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:21,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:21,232 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:21,232 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:21,232 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:21,233 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:21,233 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:21,233 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:21,233 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:21,233 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:21,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:21,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:21,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:21,233 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:21,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:21,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:21,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:21,235 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:21,235 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:21,235 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:21,235 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:21,235 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:21,236 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:21,236 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:21,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:21,236 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:21,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:21,236 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:21,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:21,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:21,239 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:21,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:21,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:21,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:21,240 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:21,240 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:21,240 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:21,240 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:21,240 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:21,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:21,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:21,241 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:21,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:21,240 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:21,241 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:21,240 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:21,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:21,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:21,241 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:21,241 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:21,241 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:21,241 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:21,241 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:21,341 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:21,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:21,343 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:21,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:21,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:21,344 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:21,344 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:21,344 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:21,344 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:21,344 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:21,344 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:21,345 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:21,345 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:21,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:21,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:21,345 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:21,345 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:21,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:21,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:21,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:21,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:21,346 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:21,346 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:21,346 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:21,346 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:21,346 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:21,346 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:21,346 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:21,346 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:21,346 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:21,346 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:21,346 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0013c9607e8a284ca05e07f23e7866be 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-08 20:57:21,347 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=A 2023-04-08 20:57:21,347 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:21,347 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=B 2023-04-08 20:57:21,347 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:21,347 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=C 2023-04-08 20:57:21,347 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:21,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 0013c9607e8a284ca05e07f23e7866be 2023-04-08 20:57:21,541 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:21,559 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:21,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987501552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:21,563 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:21,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987501558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:21,567 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:21,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987501559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:21,663 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:21,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987501660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:21,666 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:21,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987501663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:21,672 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:21,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987501668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:21,752 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=273 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/91b6dfc00d844addba3538824a795d23 2023-04-08 20:57:21,869 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:21,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987501865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:21,870 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:21,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987501867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:21,875 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:21,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987501873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,161 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=273 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/6100dafa9531443b8b59029bbff23424 2023-04-08 20:57:22,173 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:22,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987502170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:22,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987502171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,180 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:22,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987502177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,568 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=273 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/1d2c79ef01d043d0bebf2939ae8fa32f 2023-04-08 20:57:22,571 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/91b6dfc00d844addba3538824a795d23 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/91b6dfc00d844addba3538824a795d23 2023-04-08 20:57:22,574 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/91b6dfc00d844addba3538824a795d23, entries=150, sequenceid=273, filesize=11.9 K 2023-04-08 20:57:22,574 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/6100dafa9531443b8b59029bbff23424 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/6100dafa9531443b8b59029bbff23424 2023-04-08 20:57:22,577 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/6100dafa9531443b8b59029bbff23424, entries=150, sequenceid=273, filesize=11.9 K 2023-04-08 20:57:22,577 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/1d2c79ef01d043d0bebf2939ae8fa32f as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/1d2c79ef01d043d0bebf2939ae8fa32f 2023-04-08 20:57:22,580 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/1d2c79ef01d043d0bebf2939ae8fa32f, entries=150, sequenceid=273, filesize=11.9 K 2023-04-08 20:57:22,580 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 0013c9607e8a284ca05e07f23e7866be in 1234ms, sequenceid=273, compaction requested=true 2023-04-08 20:57:22,580 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:22,580 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:22,580 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:22,580 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:22,580 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,581 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:22,581 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:22,581 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:22,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:22,581 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:22,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:22,582 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:22,582 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:22,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:22,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:22,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:22,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:22,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a68799[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:22,583 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:22,583 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:22,583 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:22,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:22,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:22,583 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:22,583 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:22,583 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,583 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:22,583 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:22,584 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:22,584 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,584 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:22,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:22,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:22,584 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:22,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:22,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:22,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:22,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:22,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:22,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:22,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:22,586 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:22,586 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:22,586 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:22,586 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:22,586 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:22,587 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:22,587 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:22,587 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:22,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:22,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:22,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:22,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:22,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:22,587 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:22,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:22,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:22,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:22,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:22,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:22,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:22,588 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:22,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:22,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,590 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,590 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:22,590 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:22,590 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:22,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:22,590 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:22,590 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:22,590 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:22,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:22,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:22,590 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:22,590 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,591 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:22,591 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:22,591 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:22,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:22,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:22,591 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:22,591 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:22,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 0013c9607e8a284ca05e07f23e7866be 2023-04-08 20:57:22,681 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0013c9607e8a284ca05e07f23e7866be 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-08 20:57:22,681 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=A 2023-04-08 20:57:22,681 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:22,681 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=B 2023-04-08 20:57:22,681 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:22,681 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=C 2023-04-08 20:57:22,681 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:22,689 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/fe79324a6a174137a15accb76b5080cc 2023-04-08 20:57:22,691 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:22,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:22,693 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:22,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:22,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:22,694 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:22,694 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:22,694 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:22,694 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:22,695 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:22,695 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:22,696 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:22,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:22,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:22,696 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:22,696 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,696 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:22,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:22,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:22,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:22,696 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:22,696 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:22,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:22,697 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:22,697 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:22,697 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:22,697 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:22,697 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:22,697 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:22,697 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:22,697 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:22,697 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:22,697 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:22,697 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:22,697 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,697 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:22,697 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:22,698 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:22,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:22,698 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:22,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:22,698 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:22,698 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:22,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:22,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:22,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:22,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:22,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50e1e377[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:22,699 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:22,699 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:22,700 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:22,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:22,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:22,700 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:22,700 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:22,700 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,700 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:22,700 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:22,700 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,700 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:22,700 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:22,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:22,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:22,700 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:22,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:22,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:22,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:22,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:22,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:22,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:22,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:22,702 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:22,702 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:22,702 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:22,702 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:22,702 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:22,703 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:22,703 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:22,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:22,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:22,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:22,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:22,703 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:22,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:22,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:22,703 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:22,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:22,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:22,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:22,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:22,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:22,704 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:22,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:22,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,705 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,705 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:22,705 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:22,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:22,705 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:22,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:22,705 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:22,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:22,705 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:22,705 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:22,706 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:22,705 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:22,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:22,706 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:22,706 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:22,706 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:22,706 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:22,706 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:22,736 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:22,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987502726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,736 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:22,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987502727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,737 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:22,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987502728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,806 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:22,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:22,808 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:22,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:22,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:22,809 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:22,809 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:22,809 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:22,809 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:22,809 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:22,809 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:22,810 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:22,810 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:22,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:22,810 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,810 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:22,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:22,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:22,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:22,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:22,811 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:22,811 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:22,811 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:22,811 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:22,811 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:22,811 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:22,811 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:22,811 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:22,811 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:22,811 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:22,811 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:22,811 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:22,811 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:22,811 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:22,811 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,811 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:22,811 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:22,812 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:22,812 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:22,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:22,812 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:22,812 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:22,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:22,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:22,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:22,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:22,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:22,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6696426f[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:22,813 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:22,813 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:22,814 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:22,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:22,814 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:22,814 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,814 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:22,814 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:22,814 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:22,815 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,815 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:22,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:22,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:22,815 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:22,815 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:22,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:22,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:22,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:22,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:22,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:22,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:22,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:22,816 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:22,816 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:22,816 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:22,816 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:22,816 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:22,817 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:22,817 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:22,817 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:22,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:22,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:22,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:22,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:22,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:22,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:22,818 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:22,818 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:22,820 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,820 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:22,820 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:22,820 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:22,820 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:22,820 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:22,820 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:22,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:22,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:22,820 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:22,820 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:22,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:22,820 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:22,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:22,820 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:22,820 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:22,820 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:22,820 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:22,841 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:22,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987502837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,843 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:22,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987502837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,843 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:22,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987502838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,927 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:22,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:22,929 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:22,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:22,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:22,930 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:22,930 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:22,930 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:22,930 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:22,931 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:22,931 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:22,931 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,931 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:22,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:22,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:22,931 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,932 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:22,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:22,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:22,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:22,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:22,932 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:22,932 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:22,932 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:22,932 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:22,932 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:22,932 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:22,932 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:22,932 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:22,932 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:22,932 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:22,933 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:22,933 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:22,933 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:22,933 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,933 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:22,933 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:22,933 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,933 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:22,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:22,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:22,934 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:22,934 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:22,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:22,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:22,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:22,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:22,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:22,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74ade8c7[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:22,935 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:22,935 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:22,935 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:22,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:22,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:22,935 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:22,935 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:22,935 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,935 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:22,935 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:22,936 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:22,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:22,936 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:22,936 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:22,936 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:22,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:22,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:22,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:22,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:22,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:22,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:22,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:22,938 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:22,938 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:22,938 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:22,938 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:22,938 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:22,938 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:22,938 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:22,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:22,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:22,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:22,939 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:22,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:22,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:22,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:22,939 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:22,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:22,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:22,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:22,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:22,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:22,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,940 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:22,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:22,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,943 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:22,943 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:22,943 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:22,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:22,943 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:22,943 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:22,943 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:22,943 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:22,943 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:22,943 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:22,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:22,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:22,944 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:22,944 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:22,944 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:22,944 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:22,944 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:23,044 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:23,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:23,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:23,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987503042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,048 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:23,049 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:23,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987503044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:23,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:23,049 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:23,049 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:23,049 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:23,049 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:23,049 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,050 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:23,050 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:23,050 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:23,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:23,050 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,050 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:23,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:23,051 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:23,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987503045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:23,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:23,051 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:23,052 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:23,052 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:23,052 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:23,052 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:23,052 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:23,052 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:23,052 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:23,052 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:23,052 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:23,052 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:23,052 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:23,052 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:23,052 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:23,052 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,052 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:23,052 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:23,054 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,054 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:23,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:23,054 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,054 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:23,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:23,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:23,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:23,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:23,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:23,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44873384[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:23,055 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:23,055 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,056 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,056 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:23,056 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:23,056 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,056 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:23,056 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:23,056 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,056 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:23,057 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:23,057 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:23,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:23,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:23,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:23,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:23,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:23,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:23,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:23,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:23,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:23,058 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:23,058 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:23,058 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:23,058 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:23,058 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:23,059 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,059 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:23,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:23,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,059 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:23,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:23,060 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:23,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:23,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:23,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:23,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:23,060 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:23,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:23,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,062 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,062 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:23,062 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:23,062 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:23,062 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:23,062 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:23,062 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,062 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,062 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,062 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:23,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:23,062 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:23,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:23,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:23,062 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:23,062 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:23,062 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:23,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:23,096 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/64aedb613dad451090529a0c6315db60 2023-04-08 20:57:23,162 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:23,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:23,164 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:23,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:23,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:23,165 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:23,165 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:23,165 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:23,165 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:23,165 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,165 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:23,166 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,166 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:23,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:23,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:23,166 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,166 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:23,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:23,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:23,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:23,166 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:23,167 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:23,167 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:23,167 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:23,167 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:23,167 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:23,167 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:23,167 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:23,167 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:23,167 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:23,167 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:23,167 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:23,167 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:23,167 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:23,167 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,167 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:23,167 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:23,168 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,168 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:23,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:23,168 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,168 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:23,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:23,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:23,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:23,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:23,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:23,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b228a38[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:23,169 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:23,169 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,170 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,170 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:23,170 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:23,170 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:23,170 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,170 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:23,171 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,171 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:23,171 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:23,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:23,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:23,171 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:23,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:23,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:23,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:23,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:23,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:23,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:23,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:23,172 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:23,173 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:23,173 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:23,173 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:23,173 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:23,173 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:23,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:23,173 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,173 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:23,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:23,174 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:23,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:23,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:23,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:23,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:23,175 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:23,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:23,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,176 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,176 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:23,176 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:23,176 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:23,176 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:23,176 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:23,176 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:23,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:23,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:23,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:23,176 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,176 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:23,176 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:23,176 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,176 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:23,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:23,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:23,176 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,176 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,276 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:23,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:23,278 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:23,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:23,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:23,280 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:23,280 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:23,280 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:23,280 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:23,280 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,280 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:23,281 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,281 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:23,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:23,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:23,281 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,281 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:23,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:23,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:23,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:23,282 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:23,282 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:23,282 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:23,282 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:23,282 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:23,282 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:23,282 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:23,282 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:23,282 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:23,282 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:23,283 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:23,283 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:23,283 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:23,283 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:23,283 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,283 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:23,283 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:23,284 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,284 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:23,284 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,284 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:23,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:23,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f732609[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:23,285 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:23,285 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,285 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,285 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:23,285 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,286 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:23,286 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:23,286 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:23,286 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:23,286 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,286 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:23,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:23,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:23,286 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:23,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:23,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:23,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:23,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:23,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:23,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:23,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:23,288 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:23,288 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:23,288 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:23,288 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:23,288 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:23,289 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,289 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:23,289 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:23,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:23,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:23,289 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:23,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:23,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:23,290 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:23,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,291 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,292 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:23,292 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:23,292 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:23,292 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:23,292 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:23,292 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,292 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,292 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,292 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,292 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:23,292 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:23,292 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:23,292 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:23,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:23,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:23,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:23,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:23,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:23,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987503348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:23,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987503349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,357 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:23,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987503351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,392 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:23,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:23,394 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:23,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:23,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:23,394 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:23,394 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:23,394 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:23,395 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:23,395 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,395 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:23,395 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,395 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:23,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:23,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:23,396 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,396 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:23,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:23,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:23,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:23,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:23,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:23,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:23,396 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:23,397 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:23,397 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:23,397 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:23,397 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:23,397 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:23,397 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:23,397 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:23,397 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:23,397 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:23,397 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:23,397 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,397 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:23,397 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:23,398 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,398 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:23,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:23,398 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,398 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:23,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:23,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:23,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:23,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:23,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:23,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@548314f0[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:23,400 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:23,400 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,401 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,401 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:23,401 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,401 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:23,401 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:23,401 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:23,402 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,402 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:23,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:23,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:23,402 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:23,402 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:23,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:23,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:23,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:23,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:23,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:23,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:23,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:23,403 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:23,404 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:23,404 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:23,404 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:23,404 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:23,404 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:23,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:23,404 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,405 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:23,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:23,405 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:23,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:23,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:23,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:23,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:23,406 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:23,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:23,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,408 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,408 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:23,408 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:23,408 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,408 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,408 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:23,408 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:23,408 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:23,408 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:23,408 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:23,408 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,408 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:23,408 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:23,408 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:23,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:23,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:23,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:23,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:23,504 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/a2cdc2252fc04a4698cb688f5f3713dd 2023-04-08 20:57:23,507 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/fe79324a6a174137a15accb76b5080cc as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/fe79324a6a174137a15accb76b5080cc 2023-04-08 20:57:23,508 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:23,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:23,510 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/fe79324a6a174137a15accb76b5080cc, entries=200, sequenceid=286, filesize=14.3 K 2023-04-08 20:57:23,511 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/64aedb613dad451090529a0c6315db60 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/64aedb613dad451090529a0c6315db60 2023-04-08 20:57:23,512 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:23,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:23,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:23,513 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:23,513 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:23,514 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:23,514 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:23,514 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,514 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:23,515 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:23,515 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:23,515 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,515 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:23,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:23,515 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:23,516 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:23,516 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:23,516 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:23,516 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:23,516 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:23,516 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:23,516 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:23,516 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:23,516 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:23,516 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:23,516 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:23,516 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:23,516 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:23,516 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,516 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:23,516 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:23,517 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,517 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:23,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:23,517 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,517 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:23,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:23,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:23,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:23,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:23,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:23,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@296516f7[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:23,518 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:23,518 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,519 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,519 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:23,519 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:23,519 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:23,519 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,519 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:23,519 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/64aedb613dad451090529a0c6315db60, entries=150, sequenceid=286, filesize=11.9 K 2023-04-08 20:57:23,520 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/a2cdc2252fc04a4698cb688f5f3713dd as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/a2cdc2252fc04a4698cb688f5f3713dd 2023-04-08 20:57:23,520 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,520 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:23,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:23,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:23,520 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:23,520 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:23,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:23,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:23,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:23,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:23,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:23,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:23,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:23,522 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:23,522 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:23,522 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:23,522 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:23,522 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:23,522 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/a2cdc2252fc04a4698cb688f5f3713dd, entries=150, sequenceid=286, filesize=11.9 K 2023-04-08 20:57:23,523 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,523 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:23,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,523 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:23,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:23,523 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 0013c9607e8a284ca05e07f23e7866be in 842ms, sequenceid=286, compaction requested=true 2023-04-08 20:57:23,523 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:23,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:23,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:23,523 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-08 20:57:23,523 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:57:23,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,524 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-08 20:57:23,524 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:57:23,524 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:57:23,524 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:23,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:23,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:23,525 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 63194 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-08 20:57:23,525 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 0013c9607e8a284ca05e07f23e7866be/A is initiating minor compaction (all files) 2023-04-08 20:57:23,525 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0013c9607e8a284ca05e07f23e7866be/A in TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:23,525 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/ec5c3899e7d745f4b2aa5fd197533ee1, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/321999a731bb494fa12c3c5d1f8d8db8, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/1b02e15a972e44fc8b0d3d8314956966, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/91b6dfc00d844addba3538824a795d23, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/fe79324a6a174137a15accb76b5080cc] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp, totalSize=61.7 K 2023-04-08 20:57:23,525 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60754 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-08 20:57:23,525 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): 0013c9607e8a284ca05e07f23e7866be/B is initiating minor compaction (all files) 2023-04-08 20:57:23,525 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0013c9607e8a284ca05e07f23e7866be/B in TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:23,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:23,525 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/649fbd0af4e447fea3389ca1419c22e9, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/22617e11ffac47bf977e0533d1a35d39, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/496ac6ea10624e4c81265a6a0b80c79a, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/6100dafa9531443b8b59029bbff23424, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/64aedb613dad451090529a0c6315db60] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp, totalSize=59.3 K 2023-04-08 20:57:23,525 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,526 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting ec5c3899e7d745f4b2aa5fd197533ee1, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680987437471 2023-04-08 20:57:23,526 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 649fbd0af4e447fea3389ca1419c22e9, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680987437471 2023-04-08 20:57:23,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:23,526 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 321999a731bb494fa12c3c5d1f8d8db8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1680987438123 2023-04-08 20:57:23,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:23,526 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 22617e11ffac47bf977e0533d1a35d39, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1680987438123 2023-04-08 20:57:23,526 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 1b02e15a972e44fc8b0d3d8314956966, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1680987439266 2023-04-08 20:57:23,527 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 496ac6ea10624e4c81265a6a0b80c79a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1680987439266 2023-04-08 20:57:23,527 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 91b6dfc00d844addba3538824a795d23, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1680987440413 2023-04-08 20:57:23,527 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 6100dafa9531443b8b59029bbff23424, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1680987440413 2023-04-08 20:57:23,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,527 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting fe79324a6a174137a15accb76b5080cc, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1680987441540 2023-04-08 20:57:23,527 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 64aedb613dad451090529a0c6315db60, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1680987441540 2023-04-08 20:57:23,528 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:23,528 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:23,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:23,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:23,528 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,528 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:23,528 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:23,528 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,528 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,528 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:23,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:23,528 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:23,528 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:23,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:23,528 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:23,528 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:23,529 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:23,529 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:23,533 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 0013c9607e8a284ca05e07f23e7866be#A#compaction#400 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:57:23,534 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0013c9607e8a284ca05e07f23e7866be#B#compaction#401 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:57:23,540 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/4768bc1a32304346bf21511c21129d1d as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/4768bc1a32304346bf21511c21129d1d 2023-04-08 20:57:23,544 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 0013c9607e8a284ca05e07f23e7866be/B of 0013c9607e8a284ca05e07f23e7866be into 4768bc1a32304346bf21511c21129d1d(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:57:23,544 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:23,544 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be., storeName=0013c9607e8a284ca05e07f23e7866be/B, priority=11, startTime=1680987443524; duration=0sec 2023-04-08 20:57:23,544 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:57:23,544 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-08 20:57:23,546 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60754 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-08 20:57:23,546 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): 0013c9607e8a284ca05e07f23e7866be/C is initiating minor compaction (all files) 2023-04-08 20:57:23,546 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0013c9607e8a284ca05e07f23e7866be/C in TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:23,546 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/90934e93d68743d4b72b4cdeb7046d35, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/ddfd72e78aff4b7abe3ae11634a808bc, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/ad905d732fcd494981238323edb53537, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/1d2c79ef01d043d0bebf2939ae8fa32f, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/a2cdc2252fc04a4698cb688f5f3713dd] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp, totalSize=59.3 K 2023-04-08 20:57:23,546 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 90934e93d68743d4b72b4cdeb7046d35, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680987437471 2023-04-08 20:57:23,546 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting ddfd72e78aff4b7abe3ae11634a808bc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1680987438123 2023-04-08 20:57:23,547 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting ad905d732fcd494981238323edb53537, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1680987439266 2023-04-08 20:57:23,547 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 1d2c79ef01d043d0bebf2939ae8fa32f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1680987440413 2023-04-08 20:57:23,547 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting a2cdc2252fc04a4698cb688f5f3713dd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1680987441540 2023-04-08 20:57:23,555 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0013c9607e8a284ca05e07f23e7866be#C#compaction#402 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:57:23,569 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/a2028e9fe2d04b1d982332dc365341a6 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/a2028e9fe2d04b1d982332dc365341a6 2023-04-08 20:57:23,573 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 0013c9607e8a284ca05e07f23e7866be/C of 0013c9607e8a284ca05e07f23e7866be into a2028e9fe2d04b1d982332dc365341a6(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:57:23,573 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:23,573 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be., storeName=0013c9607e8a284ca05e07f23e7866be/C, priority=11, startTime=1680987443524; duration=0sec 2023-04-08 20:57:23,573 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:57:23,629 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:23,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:23,631 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:23,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:23,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:23,631 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:23,631 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:23,631 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:23,631 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:23,632 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,632 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:23,632 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,632 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:23,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:23,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:23,633 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,633 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:23,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:23,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:23,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:23,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:23,633 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:23,633 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:23,633 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:23,634 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:23,634 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:23,634 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:23,634 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:23,634 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:23,634 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:23,634 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:23,634 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0013c9607e8a284ca05e07f23e7866be 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-08 20:57:23,634 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=A 2023-04-08 20:57:23,634 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:23,634 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=B 2023-04-08 20:57:23,634 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:23,634 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=C 2023-04-08 20:57:23,635 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:23,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 0013c9607e8a284ca05e07f23e7866be 2023-04-08 20:57:23,861 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:23,876 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:23,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987503871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,879 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:23,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987503873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,879 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:23,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987503874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,940 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/890f2082a0534325a50128ede792bf7b as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/890f2082a0534325a50128ede792bf7b 2023-04-08 20:57:23,943 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 0013c9607e8a284ca05e07f23e7866be/A of 0013c9607e8a284ca05e07f23e7866be into 890f2082a0534325a50128ede792bf7b(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:57:23,943 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:23,943 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be., storeName=0013c9607e8a284ca05e07f23e7866be/A, priority=11, startTime=1680987443523; duration=0sec 2023-04-08 20:57:23,943 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:57:23,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:23,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987503978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,981 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:23,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987503979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:23,984 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:23,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987503981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:24,041 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=307 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/5baa9793369640548636e2354be557f0 2023-04-08 20:57:24,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:24,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987504181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:24,187 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:24,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987504182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:24,188 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:24,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987504185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:24,449 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=307 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/962a58af2aea4dbea5c1efd9fa32412a 2023-04-08 20:57:24,487 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:24,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987504485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:24,491 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:24,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987504488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:24,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:24,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987504490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:24,857 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=307 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/644c38b4e2854999975e07e9f21d8705 2023-04-08 20:57:24,860 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/5baa9793369640548636e2354be557f0 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/5baa9793369640548636e2354be557f0 2023-04-08 20:57:24,863 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/5baa9793369640548636e2354be557f0, entries=150, sequenceid=307, filesize=11.9 K 2023-04-08 20:57:24,864 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/962a58af2aea4dbea5c1efd9fa32412a as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/962a58af2aea4dbea5c1efd9fa32412a 2023-04-08 20:57:24,867 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/962a58af2aea4dbea5c1efd9fa32412a, entries=150, sequenceid=307, filesize=11.9 K 2023-04-08 20:57:24,868 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/644c38b4e2854999975e07e9f21d8705 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/644c38b4e2854999975e07e9f21d8705 2023-04-08 20:57:24,871 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/644c38b4e2854999975e07e9f21d8705, entries=150, sequenceid=307, filesize=11.9 K 2023-04-08 20:57:24,871 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 0013c9607e8a284ca05e07f23e7866be in 1237ms, sequenceid=307, compaction requested=false 2023-04-08 20:57:24,871 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:24,871 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:24,872 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:24,872 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:24,872 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:24,872 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:24,872 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:24,872 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:24,872 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:24,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:24,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:24,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:24,873 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:24,873 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:24,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:24,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:24,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:24,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:24,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:24,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:24,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26993aa9[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:24,874 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:24,874 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:24,874 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:24,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:24,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:24,875 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:24,875 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:24,875 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:24,875 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:24,875 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:24,875 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:24,875 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:24,875 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:24,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:24,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:24,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:24,875 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:24,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:24,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:24,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:24,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:24,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:24,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:24,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:24,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:24,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:24,877 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:24,877 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:24,877 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:24,877 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:24,877 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:24,878 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:24,878 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:24,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:24,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:24,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:24,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:24,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:24,878 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:24,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:24,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:24,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:24,879 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:24,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:24,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:24,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:24,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:24,880 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:24,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:24,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:24,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:24,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:24,881 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:24,881 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:24,881 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:24,881 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:24,881 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:24,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:24,881 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:24,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:24,881 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:24,881 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:24,882 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:24,882 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:24,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:24,882 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:24,882 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:24,882 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:24,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:24,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:24,882 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:24,982 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:24,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:24,983 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:24,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:24,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:24,984 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:24,984 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:24,984 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:24,984 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:24,985 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:24,985 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:24,985 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:24,986 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:24,986 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:24,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 0013c9607e8a284ca05e07f23e7866be 2023-04-08 20:57:24,993 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0013c9607e8a284ca05e07f23e7866be 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-08 20:57:24,993 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=A 2023-04-08 20:57:24,993 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:24,993 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=B 2023-04-08 20:57:24,993 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:24,993 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=C 2023-04-08 20:57:24,993 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:24,994 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:24,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:24,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:24,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:24,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:24,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:24,998 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:24,998 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:24,998 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:24,998 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:24,998 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:25,002 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:25,002 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:25,002 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:25,002 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:25,002 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:25,002 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:25,003 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:25,003 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:25,003 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,003 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:25,003 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:25,004 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,004 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:25,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:25,004 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,004 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:25,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:25,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:25,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:25,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:25,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@700c131d[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:25,006 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:25,006 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,006 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,006 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:25,006 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:25,006 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:25,007 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,007 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:25,007 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:25,007 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,007 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:25,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:25,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:25,008 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:25,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:25,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:25,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:25,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:25,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:25,010 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:25,010 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:25,010 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:25,010 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:25,010 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:25,012 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,012 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:25,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:25,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,012 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:25,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:25,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:25,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:25,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:25,014 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:25,015 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,023 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:25,023 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:25,023 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:25,023 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:25,023 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:25,023 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:25,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:25,023 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:25,023 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:25,023 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,023 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,023 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,023 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,023 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,023 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:25,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:25,033 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:25,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987505026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,035 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:25,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987505028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,035 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:25,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987505028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,123 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:25,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:25,126 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:25,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:25,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:25,127 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:25,127 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:25,127 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:25,127 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:25,127 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,127 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,128 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,128 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:25,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:25,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:25,128 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,128 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:25,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:25,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,129 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:25,129 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:25,129 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:25,129 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:25,129 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:25,129 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:25,129 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:25,129 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:25,129 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:25,129 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:25,129 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:25,129 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:25,129 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:25,129 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,129 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:25,129 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:25,131 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,131 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:25,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:25,132 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,132 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:25,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:25,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:25,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:25,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:25,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c64bd78[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:25,133 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:25,133 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,133 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,133 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:25,133 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:25,133 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:25,133 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,133 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:25,134 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:25,134 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,134 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:25,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:25,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:25,134 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:25,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:25,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:25,135 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:25,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987505134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:25,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:25,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:25,136 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:25,136 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:25,136 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:25,136 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:25,136 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:25,137 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,137 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:25,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:25,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,137 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:25,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:25,138 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:25,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:25,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:25,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,138 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:25,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,140 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,140 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,140 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,140 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,140 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,140 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,140 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:25,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:25,140 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:25,140 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:25,140 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:25,140 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,140 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:25,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:25,140 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:25,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:25,140 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:25,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:25,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:25,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:25,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987505136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:25,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987505136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,240 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:25,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:25,242 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:25,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:25,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:25,243 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:25,243 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:25,243 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:25,243 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:25,243 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,244 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,244 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,244 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:25,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:25,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:25,244 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,244 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:25,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:25,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,245 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:25,245 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:25,245 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:25,245 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:25,245 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:25,245 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:25,245 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:25,245 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:25,245 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:25,245 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:25,245 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:25,245 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:25,246 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:25,246 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:25,246 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,246 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:25,246 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:25,246 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,246 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:25,247 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:25,247 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:25,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:25,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:25,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:25,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d37d6ad[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:25,247 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:25,248 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,248 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,248 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:25,248 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:25,248 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,248 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:25,248 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:25,249 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,249 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:25,249 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:25,249 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:25,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:25,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:25,250 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:25,250 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:25,250 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:25,250 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:25,250 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:25,251 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,251 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,251 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:25,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:25,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:25,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:25,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:25,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:25,252 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:25,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,253 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:25,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,254 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,254 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,254 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:25,254 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,254 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:25,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:25,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:25,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:25,254 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:25,254 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,254 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:25,254 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:25,254 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,254 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:25,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:25,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:25,254 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:25,254 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,254 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,339 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:25,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987505336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,346 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:25,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987505342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,346 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:25,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987505342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,354 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:25,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:25,356 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:25,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:25,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:25,357 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:25,357 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:25,357 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:25,357 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:25,358 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,358 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,358 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,358 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:25,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:25,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:25,358 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,358 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:25,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:25,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,359 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:25,359 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:25,359 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:25,359 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:25,359 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:25,360 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:25,360 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:25,360 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:25,360 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:25,360 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:25,360 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:25,360 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:25,360 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:25,360 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:25,360 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,360 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:25,360 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:25,361 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:25,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:25,361 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:25,361 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,361 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:25,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:25,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:25,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:25,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5294efd0[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:25,362 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:25,362 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,363 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,363 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,363 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:25,363 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:25,363 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:25,363 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:25,364 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,364 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:25,364 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:25,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:25,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:25,364 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:25,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:25,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:25,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:25,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:25,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:25,365 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:25,365 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:25,365 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:25,365 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:25,365 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:25,366 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,366 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:25,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:25,366 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:25,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:25,366 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,367 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:25,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,369 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,369 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,369 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,369 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,369 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:25,369 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:25,369 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:25,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:25,369 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:25,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:25,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:25,369 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:25,369 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,370 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:25,370 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,370 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,370 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:25,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:25,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:25,426 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/af720d4c005342d39d23737ddaa3a171 2023-04-08 20:57:25,434 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/32edda62db114ba48b7da78a99056f3b 2023-04-08 20:57:25,470 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:25,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:25,471 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:25,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:25,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:25,472 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:25,472 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:25,472 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:25,472 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:25,473 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,473 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,474 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,474 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:25,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:25,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:25,474 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,474 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:25,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:25,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,475 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:25,475 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:25,475 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:25,475 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:25,475 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:25,475 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:25,475 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:25,475 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:25,475 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:25,476 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:25,476 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:25,476 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:25,476 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:25,476 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:25,476 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,476 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:25,476 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:25,476 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,476 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:25,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:25,477 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:25,477 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:25,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:25,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:25,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:25,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42909a64[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:25,478 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:25,478 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,478 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,479 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:25,479 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,479 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:25,479 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:25,479 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:25,480 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,480 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:25,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:25,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:25,480 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:25,480 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:25,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:25,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:25,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:25,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:25,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:25,482 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:25,482 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:25,482 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:25,482 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:25,482 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:25,483 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,483 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:25,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:25,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,483 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:25,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:25,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:25,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:25,483 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:25,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,484 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:25,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,486 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,486 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,486 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,486 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:25,486 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:25,486 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:25,486 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,486 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:25,486 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:25,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:25,486 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:25,486 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,486 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:25,486 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,486 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:25,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:25,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:25,586 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:25,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:25,588 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:25,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:25,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:25,589 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:25,589 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:25,589 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:25,589 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:25,589 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,589 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,590 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,590 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:25,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:25,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:25,590 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,590 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:25,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:25,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,590 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:25,591 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:25,591 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:25,591 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:25,591 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:25,591 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:25,591 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:25,591 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:25,591 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:25,591 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:25,591 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:25,591 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:25,591 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:25,591 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:25,591 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,591 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:25,591 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:25,592 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,592 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:25,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:25,592 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,592 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:25,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:25,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:25,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:25,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:25,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b866033[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:25,593 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:25,593 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,594 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,594 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:25,594 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,594 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:25,594 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:25,594 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:25,595 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:25,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:25,595 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:25,595 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:25,595 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:25,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:25,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:25,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:25,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:25,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:25,597 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:25,597 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:25,597 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:25,597 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:25,597 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:25,598 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,598 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:25,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:25,598 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:25,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:25,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:25,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:25,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,598 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:25,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,599 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:25,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,600 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,600 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,600 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,600 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:25,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:25,600 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:25,601 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:25,600 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:25,600 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:25,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:25,601 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:25,601 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:25,601 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,601 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,601 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,601 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:25,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:25,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:25,640 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:25,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987505640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,654 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:25,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987505647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,654 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:25,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987505649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,701 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:25,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:25,703 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:25,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:25,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:25,704 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:25,704 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:25,704 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:25,704 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:25,704 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,704 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,705 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,705 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:25,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:25,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:25,705 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,705 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:25,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:25,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,706 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:25,706 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:25,706 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:25,706 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:25,706 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:25,706 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:25,706 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:25,706 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:25,706 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:25,706 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:25,706 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:25,706 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:25,706 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:25,706 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:25,706 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,706 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:25,706 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:25,707 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,707 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:25,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:25,707 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,707 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:25,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:25,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:25,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:25,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:25,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58874750[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:25,708 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:25,708 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,709 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,709 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:25,709 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:25,709 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,709 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:25,709 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:25,710 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,710 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:25,710 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:25,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:25,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:25,710 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:25,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:25,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:25,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:25,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:25,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:25,711 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:25,711 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:25,711 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:25,711 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:25,711 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:25,712 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,712 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:25,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,712 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:25,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:25,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:25,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:25,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:25,712 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:25,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,713 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:25,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,715 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,715 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:25,715 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,715 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:25,715 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,715 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:25,715 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:25,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:25,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:25,715 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:25,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:25,715 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:25,715 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,715 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:25,716 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,716 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,716 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:25,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:25,816 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:25,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:25,817 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:25,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:25,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:25,818 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:25,818 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:25,819 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:25,819 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:25,819 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,819 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,820 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:25,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:25,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:25,820 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,820 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,820 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:25,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:25,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,821 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:25,821 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:25,821 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:25,821 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:25,821 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:25,821 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:25,821 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:25,821 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:25,821 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:25,821 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:25,821 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:25,821 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:25,821 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:25,821 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:25,821 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,821 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:25,821 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:25,822 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,822 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:25,822 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,822 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:25,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:25,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:25,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:25,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:25,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d18c9ff[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:25,824 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:25,824 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,824 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,824 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:25,824 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:25,824 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:25,824 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,824 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:25,825 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,825 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:25,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:25,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:25,825 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:25,825 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:25,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:25,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:25,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:25,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:25,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:25,827 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:25,827 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:25,827 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:25,827 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:25,827 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:25,828 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,828 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:25,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:25,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,828 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:25,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:25,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:25,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:25,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,829 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:25,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,830 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:25,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:25,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,831 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,832 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,832 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:25,832 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,832 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,832 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:25,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:25,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:25,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:25,832 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,832 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:25,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:25,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:25,832 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:25,832 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:25,832 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:25,832 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,832 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:25,832 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:25,842 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/3ab1eca87c064fae872ec6aebc9f76ce 2023-04-08 20:57:25,848 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/af720d4c005342d39d23737ddaa3a171 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/af720d4c005342d39d23737ddaa3a171 2023-04-08 20:57:25,851 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/af720d4c005342d39d23737ddaa3a171, entries=200, sequenceid=322, filesize=14.3 K 2023-04-08 20:57:25,852 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/32edda62db114ba48b7da78a99056f3b as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/32edda62db114ba48b7da78a99056f3b 2023-04-08 20:57:25,855 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/32edda62db114ba48b7da78a99056f3b, entries=150, sequenceid=322, filesize=11.9 K 2023-04-08 20:57:25,855 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/3ab1eca87c064fae872ec6aebc9f76ce as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/3ab1eca87c064fae872ec6aebc9f76ce 2023-04-08 20:57:25,858 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/3ab1eca87c064fae872ec6aebc9f76ce, entries=150, sequenceid=322, filesize=11.9 K 2023-04-08 20:57:25,858 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 0013c9607e8a284ca05e07f23e7866be in 865ms, sequenceid=322, compaction requested=true 2023-04-08 20:57:25,858 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:25,858 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:57:25,858 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:57:25,858 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:57:25,858 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:57:25,858 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:57:25,859 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39442 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:57:25,859 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): 0013c9607e8a284ca05e07f23e7866be/A is initiating minor compaction (all files) 2023-04-08 20:57:25,859 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37002 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:57:25,859 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0013c9607e8a284ca05e07f23e7866be/A in TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:25,859 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 0013c9607e8a284ca05e07f23e7866be/B is initiating minor compaction (all files) 2023-04-08 20:57:25,859 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0013c9607e8a284ca05e07f23e7866be/B in TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:25,859 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/890f2082a0534325a50128ede792bf7b, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/5baa9793369640548636e2354be557f0, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/af720d4c005342d39d23737ddaa3a171] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp, totalSize=38.5 K 2023-04-08 20:57:25,859 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/4768bc1a32304346bf21511c21129d1d, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/962a58af2aea4dbea5c1efd9fa32412a, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/32edda62db114ba48b7da78a99056f3b] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp, totalSize=36.1 K 2023-04-08 20:57:25,860 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 890f2082a0534325a50128ede792bf7b, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1680987441540 2023-04-08 20:57:25,860 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 4768bc1a32304346bf21511c21129d1d, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1680987441540 2023-04-08 20:57:25,860 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 5baa9793369640548636e2354be557f0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1680987442715 2023-04-08 20:57:25,860 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 962a58af2aea4dbea5c1efd9fa32412a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1680987442715 2023-04-08 20:57:25,860 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting af720d4c005342d39d23737ddaa3a171, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1680987443866 2023-04-08 20:57:25,860 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 32edda62db114ba48b7da78a99056f3b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1680987443866 2023-04-08 20:57:25,866 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 0013c9607e8a284ca05e07f23e7866be#B#compaction#409 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:57:25,866 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0013c9607e8a284ca05e07f23e7866be#A#compaction#410 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:57:25,932 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:25,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:25,934 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:25,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:25,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:25,935 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:25,935 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:25,935 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:25,935 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:25,935 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,935 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,936 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:25,936 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:25,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:25,936 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:25,936 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:25,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:25,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:25,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:25,937 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:25,937 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:25,937 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:25,937 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:25,937 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:25,937 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:25,937 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:25,937 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:25,937 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:25,937 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:25,937 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0013c9607e8a284ca05e07f23e7866be 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-08 20:57:25,938 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=A 2023-04-08 20:57:25,938 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:25,938 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=B 2023-04-08 20:57:25,938 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:25,938 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=C 2023-04-08 20:57:25,938 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:25,944 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/55db870507b64d91b43d16d3530500d7 2023-04-08 20:57:26,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 0013c9607e8a284ca05e07f23e7866be 2023-04-08 20:57:26,144 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:26,170 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:26,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987506166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,171 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:26,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987506168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,174 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:26,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987506170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,274 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/70d13def61f641bc9ee8df353450dd5c as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/70d13def61f641bc9ee8df353450dd5c 2023-04-08 20:57:26,276 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:26,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987506271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,277 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:26,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987506272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:26,278 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0013c9607e8a284ca05e07f23e7866be/B of 0013c9607e8a284ca05e07f23e7866be into 70d13def61f641bc9ee8df353450dd5c(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:57:26,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987506274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,278 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:26,278 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be., storeName=0013c9607e8a284ca05e07f23e7866be/B, priority=13, startTime=1680987445858; duration=0sec 2023-04-08 20:57:26,278 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:57:26,278 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:57:26,279 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37002 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:57:26,279 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 0013c9607e8a284ca05e07f23e7866be/C is initiating minor compaction (all files) 2023-04-08 20:57:26,279 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0013c9607e8a284ca05e07f23e7866be/C in TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:26,279 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/a2028e9fe2d04b1d982332dc365341a6, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/644c38b4e2854999975e07e9f21d8705, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/3ab1eca87c064fae872ec6aebc9f76ce] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp, totalSize=36.1 K 2023-04-08 20:57:26,280 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting a2028e9fe2d04b1d982332dc365341a6, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1680987441540 2023-04-08 20:57:26,280 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 644c38b4e2854999975e07e9f21d8705, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1680987442715 2023-04-08 20:57:26,280 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 3ab1eca87c064fae872ec6aebc9f76ce, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1680987443866 2023-04-08 20:57:26,281 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/08772e680fc848f29593c5e8bb0b9458 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/08772e680fc848f29593c5e8bb0b9458 2023-04-08 20:57:26,285 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0013c9607e8a284ca05e07f23e7866be/A of 0013c9607e8a284ca05e07f23e7866be into 08772e680fc848f29593c5e8bb0b9458(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:57:26,285 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 0013c9607e8a284ca05e07f23e7866be#C#compaction#413 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:57:26,285 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:26,285 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be., storeName=0013c9607e8a284ca05e07f23e7866be/A, priority=13, startTime=1680987445858; duration=0sec 2023-04-08 20:57:26,285 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:57:26,293 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/06065dcaecfa490582e70ed79cbf0c66 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/06065dcaecfa490582e70ed79cbf0c66 2023-04-08 20:57:26,296 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0013c9607e8a284ca05e07f23e7866be/C of 0013c9607e8a284ca05e07f23e7866be into 06065dcaecfa490582e70ed79cbf0c66(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:57:26,296 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:26,296 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be., storeName=0013c9607e8a284ca05e07f23e7866be/C, priority=13, startTime=1680987445858; duration=0sec 2023-04-08 20:57:26,296 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:57:26,354 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/4f28b4fedb4649c994da79489ecaf893 2023-04-08 20:57:26,479 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:26,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987506477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,481 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:26,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987506478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:26,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987506480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,761 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/2ffaaf12669747d094f66835546a82b0 2023-04-08 20:57:26,764 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/55db870507b64d91b43d16d3530500d7 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/55db870507b64d91b43d16d3530500d7 2023-04-08 20:57:26,766 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/55db870507b64d91b43d16d3530500d7, entries=150, sequenceid=340, filesize=11.9 K 2023-04-08 20:57:26,767 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/4f28b4fedb4649c994da79489ecaf893 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/4f28b4fedb4649c994da79489ecaf893 2023-04-08 20:57:26,770 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/4f28b4fedb4649c994da79489ecaf893, entries=150, sequenceid=340, filesize=11.9 K 2023-04-08 20:57:26,770 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/2ffaaf12669747d094f66835546a82b0 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/2ffaaf12669747d094f66835546a82b0 2023-04-08 20:57:26,773 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/2ffaaf12669747d094f66835546a82b0, entries=150, sequenceid=340, filesize=11.9 K 2023-04-08 20:57:26,774 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 0013c9607e8a284ca05e07f23e7866be in 837ms, sequenceid=340, compaction requested=false 2023-04-08 20:57:26,774 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:26,774 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:26,774 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:26,774 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:26,774 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,775 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:26,775 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:26,776 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:26,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:26,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:26,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:26,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:26,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:26,777 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:26,778 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:26,778 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:26,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:26,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b9d3582[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:26,778 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:26,778 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:26,779 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:26,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:26,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:26,779 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:26,779 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:26,779 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:26,779 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,779 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:26,780 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,780 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:26,780 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:26,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:26,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:26,780 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:26,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:26,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:26,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:26,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:26,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:26,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:26,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:26,782 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:26,782 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:26,782 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:26,782 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:26,782 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:26,783 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:26,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:26,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:26,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:26,783 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:26,783 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:26,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:26,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:26,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:26,783 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:26,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:26,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:26,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:26,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 0013c9607e8a284ca05e07f23e7866be 2023-04-08 20:57:26,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:26,784 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0013c9607e8a284ca05e07f23e7866be 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-08 20:57:26,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:26,785 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:26,785 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=A 2023-04-08 20:57:26,785 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:26,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:26,785 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=B 2023-04-08 20:57:26,786 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:26,786 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=C 2023-04-08 20:57:26,786 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:26,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,786 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,787 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:26,787 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:26,787 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:26,787 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:26,787 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:26,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:26,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:26,787 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:26,787 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:26,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:26,787 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:26,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:26,787 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:26,787 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:26,787 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:26,787 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:26,787 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:26,815 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:26,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987506810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,816 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:26,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987506810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,817 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:26,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987506811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,887 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:26,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:26,889 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:26,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:26,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:26,890 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:26,890 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:26,890 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:26,890 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:26,890 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:26,890 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:26,891 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:26,891 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:26,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:26,891 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,891 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:26,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:26,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:26,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:26,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:26,892 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:26,892 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:26,892 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:26,892 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:26,892 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:26,892 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:26,892 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:26,892 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:26,892 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:26,892 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:26,892 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:26,892 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:26,893 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:26,893 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:26,893 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,893 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:26,893 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:26,893 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,893 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:26,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:26,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:26,894 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:26,894 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:26,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:26,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:26,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:26,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:26,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:26,895 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:26,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f2573a[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:26,895 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:26,895 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:26,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:26,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:26,895 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:26,895 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:26,895 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:26,895 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,895 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:26,896 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,896 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:26,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:26,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:26,896 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:26,896 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:26,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:26,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:26,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:26,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:26,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:26,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:26,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:26,898 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:26,898 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:26,898 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:26,898 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:26,898 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:26,906 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:26,906 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:26,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:26,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:26,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:26,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:26,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:26,906 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:26,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:26,906 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:26,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:26,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:26,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:26,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:26,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:26,907 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:26,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:26,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,909 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:26,909 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:26,909 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,909 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:26,909 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:26,909 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:26,909 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:26,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:26,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:26,909 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,909 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:26,909 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:26,909 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:26,909 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:26,909 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:26,910 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:26,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:26,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:26,920 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:26,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987506916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:26,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987506917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:26,922 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:26,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987506918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,010 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:27,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:27,011 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:27,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:27,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:27,012 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:27,012 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:27,012 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:27,012 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:27,012 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,012 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:27,014 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:27,014 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:27,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:27,014 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,014 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:27,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:27,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:27,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:27,014 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:27,015 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:27,015 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:27,015 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:27,015 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:27,015 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:27,015 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:27,015 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:27,015 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:27,015 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:27,015 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:27,015 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:27,015 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:27,015 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:27,015 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,015 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:27,015 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:27,016 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,016 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:27,016 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,016 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:27,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:27,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:27,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:27,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@612d2dd3[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:27,017 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:27,017 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,018 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,018 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:27,018 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:27,018 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,018 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:27,018 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:27,019 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,019 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:27,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:27,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:27,019 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:27,019 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:27,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:27,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:27,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:27,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:27,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:27,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:27,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:27,020 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:27,020 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:27,020 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:27,020 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:27,020 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:27,021 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:27,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:27,021 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,021 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:27,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:27,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:27,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:27,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:27,022 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:27,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:27,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,023 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:27,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:27,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,024 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,024 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:27,024 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:27,024 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:27,025 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:27,025 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:27,024 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:27,025 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:27,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:27,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:27,024 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,025 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,025 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,025 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:27,025 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,025 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:27,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:27,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:27,124 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:27,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987507123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,124 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:27,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987507123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,125 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:27,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:27,127 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:27,127 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:27,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987507124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:27,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:27,128 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:27,128 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:27,128 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:27,128 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:27,128 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,128 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:27,129 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,129 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:27,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:27,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:27,129 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,129 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:27,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:27,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:27,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:27,130 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:27,130 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:27,130 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:27,130 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:27,130 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:27,130 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:27,130 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:27,130 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:27,130 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:27,130 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:27,130 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:27,130 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:27,131 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:27,131 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:27,131 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,131 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:27,131 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:27,131 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,131 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:27,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:27,132 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,132 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:27,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:27,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:27,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:27,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:27,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:27,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a914738[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:27,133 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:27,133 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,133 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,133 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:27,134 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:27,134 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,134 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:27,134 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:27,134 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:27,134 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,134 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:27,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:27,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:27,134 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:27,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:27,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:27,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:27,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:27,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:27,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:27,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:27,136 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:27,136 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:27,136 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:27,136 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:27,136 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:27,137 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,137 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:27,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:27,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,137 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:27,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:27,137 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:27,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:27,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:27,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:27,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:27,138 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:27,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:27,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,140 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:27,140 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:27,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:27,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:27,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:27,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:27,141 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,141 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:27,141 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:27,141 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,141 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,141 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:27,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:27,141 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:27,141 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,141 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:27,142 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,142 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:27,142 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:27,195 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/9381f4a531b34b4c952cae624f73a919 2023-04-08 20:57:27,203 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/efd5bcc5cac24c5cb1c7288f7d7b6665 2023-04-08 20:57:27,242 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:27,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:27,244 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:27,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:27,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:27,244 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:27,244 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:27,244 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:27,244 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:27,245 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,245 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:27,245 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:27,245 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:27,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:27,246 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,246 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:27,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:27,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:27,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:27,246 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:27,246 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:27,246 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:27,246 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:27,246 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:27,247 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:27,247 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:27,247 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:27,247 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:27,247 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:27,247 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:27,247 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:27,247 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:27,247 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,247 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:27,247 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:27,248 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,248 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:27,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:27,248 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,248 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:27,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:27,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:27,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:27,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:27,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:27,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8a242cd[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:27,249 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:27,249 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,249 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,250 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:27,250 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:27,250 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,250 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:27,250 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:27,250 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,250 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:27,250 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:27,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:27,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:27,250 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:27,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:27,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:27,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:27,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:27,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:27,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:27,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:27,252 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:27,252 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:27,252 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:27,252 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:27,252 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:27,253 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,253 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:27,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,253 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:27,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:27,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:27,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:27,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:27,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,254 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:27,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:27,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:27,255 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:27,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:27,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,257 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:27,257 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:27,257 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:27,257 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:27,257 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:27,257 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:27,257 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:27,257 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:27,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:27,257 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:27,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:27,257 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,257 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:27,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:27,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:27,257 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,258 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,258 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,358 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:27,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:27,359 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:27,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:27,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:27,360 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:27,360 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:27,360 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:27,360 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:27,360 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,361 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:27,361 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:27,361 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:27,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:27,361 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,362 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:27,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:27,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:27,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:27,362 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:27,362 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:27,362 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:27,362 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:27,362 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:27,362 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:27,363 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:27,363 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:27,363 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:27,363 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:27,363 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:27,363 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:27,363 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:27,363 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:27,363 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,363 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:27,363 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:27,364 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,364 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:27,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:27,364 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,364 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:27,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:27,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:27,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:27,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:27,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:27,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19789f64[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:27,365 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:27,365 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,366 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,366 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:27,366 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,366 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:27,366 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:27,366 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:27,367 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:27,367 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:27,367 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,367 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:27,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:27,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:27,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:27,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:27,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:27,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:27,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:27,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:27,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:27,369 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:27,369 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:27,369 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:27,369 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:27,369 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:27,371 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,371 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:27,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:27,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,371 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:27,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:27,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:27,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:27,371 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:27,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:27,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:27,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,372 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:27,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:27,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,373 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,374 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:27,374 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:27,374 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:27,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:27,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:27,374 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:27,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:27,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:27,375 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,375 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,375 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:27,375 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:27,375 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:27,375 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,375 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,375 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:27,375 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:27,428 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:27,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987507425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,430 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:27,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987507426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,432 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:27,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987507429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,475 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:27,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:27,477 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:27,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:27,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:27,478 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:27,478 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:27,478 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:27,478 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:27,478 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,478 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:27,479 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,479 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:27,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:27,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:27,479 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,479 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:27,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:27,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:27,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:27,480 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:27,480 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:27,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:27,480 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:27,480 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:27,480 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:27,480 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:27,480 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:27,480 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:27,480 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:27,480 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:27,480 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:27,480 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:27,480 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:27,480 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,480 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:27,480 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:27,481 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,481 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:27,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:27,481 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,481 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:27,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:27,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:27,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:27,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:27,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:27,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11336a2d[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:27,482 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:27,483 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,483 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,483 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:27,483 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:27,483 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:27,483 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,483 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:27,484 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:27,484 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:27,484 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,484 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:27,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:27,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:27,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:27,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:27,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:27,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:27,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:27,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:27,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:27,486 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:27,486 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:27,486 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:27,486 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:27,486 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:27,487 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,487 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,487 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:27,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:27,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:27,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:27,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:27,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:27,487 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:27,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:27,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:27,488 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:27,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:27,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,489 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,489 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:27,489 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:27,489 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,490 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,489 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:27,490 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:27,490 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:27,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:27,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:27,490 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:27,490 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:27,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:27,490 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:27,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:27,490 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,490 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:27,490 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:27,590 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:27,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:27,592 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:27,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:27,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:27,593 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:27,593 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:27,593 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:27,593 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:27,593 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,593 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:27,594 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,594 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:27,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:27,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:27,594 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,594 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:27,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:27,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:27,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:27,595 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:27,595 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:27,595 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:27,595 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:27,595 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:27,595 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:27,595 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:27,595 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:27,595 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:27,595 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:27,595 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:27,595 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:27,595 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:27,595 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:27,595 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,595 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:27,595 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:27,596 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,596 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:27,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:27,596 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,596 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:27,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:27,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:27,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:27,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:27,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:27,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@119b47b4[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:27,598 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:27,598 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,598 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,598 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:27,598 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:27,598 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:27,598 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,598 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:27,599 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,599 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:27,599 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:27,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:27,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:27,599 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:27,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:27,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:27,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:27,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:27,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:27,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:27,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:27,601 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:27,601 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:27,601 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:27,601 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:27,601 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:27,602 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,602 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:27,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,602 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:27,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:27,602 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:27,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:27,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:27,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:27,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:27,603 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:27,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:27,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:27,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,604 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:27,604 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:27,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:27,604 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:27,604 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,604 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:27,604 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:27,604 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:27,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:27,604 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:27,605 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,605 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,605 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:27,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:27,605 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:27,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:27,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:27,605 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:27,605 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,610 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/e40267049e614350ae8d901d067c1e2b 2023-04-08 20:57:27,613 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/9381f4a531b34b4c952cae624f73a919 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/9381f4a531b34b4c952cae624f73a919 2023-04-08 20:57:27,616 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/9381f4a531b34b4c952cae624f73a919, entries=150, sequenceid=359, filesize=11.9 K 2023-04-08 20:57:27,616 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/efd5bcc5cac24c5cb1c7288f7d7b6665 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/efd5bcc5cac24c5cb1c7288f7d7b6665 2023-04-08 20:57:27,619 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/efd5bcc5cac24c5cb1c7288f7d7b6665, entries=150, sequenceid=359, filesize=11.9 K 2023-04-08 20:57:27,619 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/e40267049e614350ae8d901d067c1e2b as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/e40267049e614350ae8d901d067c1e2b 2023-04-08 20:57:27,621 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/e40267049e614350ae8d901d067c1e2b, entries=150, sequenceid=359, filesize=11.9 K 2023-04-08 20:57:27,622 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 0013c9607e8a284ca05e07f23e7866be in 838ms, sequenceid=359, compaction requested=true 2023-04-08 20:57:27,622 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:27,622 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:57:27,622 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:57:27,622 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:57:27,622 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:57:27,622 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:57:27,623 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:57:27,623 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:57:27,623 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): 0013c9607e8a284ca05e07f23e7866be/A is initiating minor compaction (all files) 2023-04-08 20:57:27,623 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 0013c9607e8a284ca05e07f23e7866be/B is initiating minor compaction (all files) 2023-04-08 20:57:27,623 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0013c9607e8a284ca05e07f23e7866be/A in TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:27,623 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0013c9607e8a284ca05e07f23e7866be/B in TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:27,623 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/08772e680fc848f29593c5e8bb0b9458, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/55db870507b64d91b43d16d3530500d7, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/9381f4a531b34b4c952cae624f73a919] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp, totalSize=36.2 K 2023-04-08 20:57:27,623 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/70d13def61f641bc9ee8df353450dd5c, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/4f28b4fedb4649c994da79489ecaf893, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/efd5bcc5cac24c5cb1c7288f7d7b6665] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp, totalSize=36.2 K 2023-04-08 20:57:27,623 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 08772e680fc848f29593c5e8bb0b9458, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1680987443866 2023-04-08 20:57:27,623 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 70d13def61f641bc9ee8df353450dd5c, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1680987443866 2023-04-08 20:57:27,624 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 55db870507b64d91b43d16d3530500d7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1680987445024 2023-04-08 20:57:27,624 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 4f28b4fedb4649c994da79489ecaf893, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1680987445024 2023-04-08 20:57:27,624 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 9381f4a531b34b4c952cae624f73a919, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1680987446162 2023-04-08 20:57:27,624 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting efd5bcc5cac24c5cb1c7288f7d7b6665, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1680987446162 2023-04-08 20:57:27,628 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 0013c9607e8a284ca05e07f23e7866be#B#compaction#418 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:57:27,630 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0013c9607e8a284ca05e07f23e7866be#A#compaction#419 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:57:27,705 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:27,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:27,707 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:27,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:27,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:27,708 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:27,708 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:27,708 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:27,708 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:27,708 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,708 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:27,709 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:27,709 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:27,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:27,709 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:27,709 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:27,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:27,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:27,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:27,710 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:27,710 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:27,710 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:27,710 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:27,710 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:27,710 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:27,710 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:27,710 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:27,710 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:27,710 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:27,710 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0013c9607e8a284ca05e07f23e7866be 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-08 20:57:27,711 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=A 2023-04-08 20:57:27,711 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:27,711 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=B 2023-04-08 20:57:27,711 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:27,711 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=C 2023-04-08 20:57:27,711 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:27,726 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/ddbc3fad0d5940c5918728cd0f47bc5d 2023-04-08 20:57:27,733 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/794e8162957940acbb9b658a336a85fb 2023-04-08 20:57:27,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 0013c9607e8a284ca05e07f23e7866be 2023-04-08 20:57:27,936 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:27,966 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:27,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987507961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,966 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:27,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987507963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:27,971 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:27,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987507963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,038 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/63f5fa9988714ead91d61e2acb6f0d43 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/63f5fa9988714ead91d61e2acb6f0d43 2023-04-08 20:57:28,043 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/4e752c66f00848febab4caec61af74e6 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/4e752c66f00848febab4caec61af74e6 2023-04-08 20:57:28,043 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0013c9607e8a284ca05e07f23e7866be/B of 0013c9607e8a284ca05e07f23e7866be into 63f5fa9988714ead91d61e2acb6f0d43(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:57:28,043 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:28,043 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be., storeName=0013c9607e8a284ca05e07f23e7866be/B, priority=13, startTime=1680987447622; duration=0sec 2023-04-08 20:57:28,043 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:57:28,043 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:57:28,044 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:57:28,044 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 0013c9607e8a284ca05e07f23e7866be/C is initiating minor compaction (all files) 2023-04-08 20:57:28,044 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0013c9607e8a284ca05e07f23e7866be/C in TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:28,044 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/06065dcaecfa490582e70ed79cbf0c66, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/2ffaaf12669747d094f66835546a82b0, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/e40267049e614350ae8d901d067c1e2b] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp, totalSize=36.2 K 2023-04-08 20:57:28,045 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 06065dcaecfa490582e70ed79cbf0c66, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1680987443866 2023-04-08 20:57:28,045 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 2ffaaf12669747d094f66835546a82b0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1680987445024 2023-04-08 20:57:28,045 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting e40267049e614350ae8d901d067c1e2b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1680987446162 2023-04-08 20:57:28,047 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0013c9607e8a284ca05e07f23e7866be/A of 0013c9607e8a284ca05e07f23e7866be into 4e752c66f00848febab4caec61af74e6(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:57:28,047 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:28,047 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be., storeName=0013c9607e8a284ca05e07f23e7866be/A, priority=13, startTime=1680987447622; duration=0sec 2023-04-08 20:57:28,047 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:57:28,049 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 0013c9607e8a284ca05e07f23e7866be#C#compaction#423 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:57:28,063 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/fa0f6758a1b04c81bd34d0248d6d6857 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/fa0f6758a1b04c81bd34d0248d6d6857 2023-04-08 20:57:28,066 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0013c9607e8a284ca05e07f23e7866be/C of 0013c9607e8a284ca05e07f23e7866be into fa0f6758a1b04c81bd34d0248d6d6857(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:57:28,066 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:28,066 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be., storeName=0013c9607e8a284ca05e07f23e7866be/C, priority=13, startTime=1680987447622; duration=0sec 2023-04-08 20:57:28,066 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:57:28,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:28,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987508067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:28,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987508067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,074 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:28,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987508072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,142 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/9a88b91b66a642d594a857a93cd130f6 2023-04-08 20:57:28,146 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/ddbc3fad0d5940c5918728cd0f47bc5d as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/ddbc3fad0d5940c5918728cd0f47bc5d 2023-04-08 20:57:28,150 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/ddbc3fad0d5940c5918728cd0f47bc5d, entries=150, sequenceid=375, filesize=11.9 K 2023-04-08 20:57:28,151 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/794e8162957940acbb9b658a336a85fb as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/794e8162957940acbb9b658a336a85fb 2023-04-08 20:57:28,155 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/794e8162957940acbb9b658a336a85fb, entries=150, sequenceid=375, filesize=11.9 K 2023-04-08 20:57:28,155 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/9a88b91b66a642d594a857a93cd130f6 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/9a88b91b66a642d594a857a93cd130f6 2023-04-08 20:57:28,158 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/9a88b91b66a642d594a857a93cd130f6, entries=150, sequenceid=375, filesize=11.9 K 2023-04-08 20:57:28,159 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 0013c9607e8a284ca05e07f23e7866be in 449ms, sequenceid=375, compaction requested=false 2023-04-08 20:57:28,159 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:28,159 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:28,159 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:28,159 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:28,159 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,159 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:28,159 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:28,160 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,160 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:28,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:28,161 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,161 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:28,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:28,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:28,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:28,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:28,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bb5db60[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:28,167 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:28,167 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,172 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,173 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:28,173 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,173 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:28,173 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:28,173 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:28,174 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:28,174 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:28,174 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:28,174 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:28,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:28,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:28,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:28,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:28,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:28,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:28,176 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:28,176 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:28,176 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:28,176 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:28,176 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:28,177 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,177 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:28,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:28,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,177 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:28,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:28,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:28,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:28,178 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:28,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,179 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:28,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,182 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:28,182 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:28,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:28,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:28,182 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,182 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:28,182 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:28,182 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:28,182 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:28,182 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:28,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:28,182 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:28,182 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,182 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:28,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:28,182 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,182 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,182 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 0013c9607e8a284ca05e07f23e7866be 2023-04-08 20:57:28,274 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0013c9607e8a284ca05e07f23e7866be 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-08 20:57:28,275 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=A 2023-04-08 20:57:28,275 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:28,275 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=B 2023-04-08 20:57:28,275 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:28,275 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=C 2023-04-08 20:57:28,275 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:28,282 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:28,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:28,284 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:28,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:28,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:28,285 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:28,285 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:28,285 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:28,285 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:28,285 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,285 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:28,286 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,286 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:28,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:28,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:28,286 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,286 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:28,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:28,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:28,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:28,286 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:28,286 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:28,287 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:28,287 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:28,287 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:28,287 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:28,287 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:28,287 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:28,287 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:28,287 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:28,287 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:28,287 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:28,287 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:28,287 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:28,287 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,287 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:28,287 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:28,288 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:28,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:28,288 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:28,288 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,288 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:28,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:28,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:28,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:28,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56ebf6cd[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:28,289 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:28,289 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,290 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,290 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:28,290 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:28,290 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:28,295 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,295 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:28,297 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,297 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:28,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:28,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:28,297 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:28,297 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:28,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:28,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:28,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:28,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:28,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:28,299 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:28,299 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:28,299 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:28,299 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:28,299 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:28,300 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,300 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:28,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:28,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,300 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:28,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:28,300 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:28,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:28,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:28,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,301 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:28,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,303 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,303 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:28,303 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:28,303 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:28,303 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:28,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:28,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:28,303 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:28,303 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:28,303 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:28,303 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:28,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:28,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:28,303 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:28,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:28,304 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,304 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,304 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,304 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:28,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987508298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:28,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987508302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,308 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:28,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987508302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,404 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:28,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:28,405 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:28,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:28,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:28,406 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:28,406 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:28,406 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:28,406 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:28,407 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,407 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:28,407 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,407 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:28,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:28,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:28,407 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,407 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:28,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:28,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:28,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:28,408 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:28,408 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:28,408 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:28,408 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:28,408 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:28,409 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:28,409 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:28,409 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:28,409 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:28,409 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:28,409 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:28,409 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:28,409 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:28,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987508405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,409 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:28,409 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:28,409 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,409 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:28,409 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:28,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987508405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,409 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:28,410 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,410 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:28,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:28,410 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,410 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:28,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:28,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:28,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:28,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:28,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c039c3f[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:28,411 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:28,411 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,412 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,412 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:28,412 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:28,412 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:28,412 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,412 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:28,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:28,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987508408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,413 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,413 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:28,413 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:28,413 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:28,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:28,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:28,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:28,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:28,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:28,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:28,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:28,415 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:28,415 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:28,415 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:28,415 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:28,415 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:28,416 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,416 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,416 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:28,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:28,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:28,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:28,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:28,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:28,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,416 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:28,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,417 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:28,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,418 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:28,418 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,418 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:28,418 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:28,418 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:28,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:28,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:28,418 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:28,418 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:28,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:28,418 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:28,419 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:28,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:28,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:28,419 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,419 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:28,419 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,419 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,419 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,498 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:28,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60764 deadline: 1680987508495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,499 DEBUG [Thread-2751] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18176 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be., hostname=jenkins-hbase20.apache.org,41091,1680987313601, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 20:57:28,519 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:28,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:28,521 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:28,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:28,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:28,521 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:28,522 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:28,522 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:28,522 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:28,522 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,522 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:28,522 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:28,522 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:28,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:28,523 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,523 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:28,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:28,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:28,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:28,524 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:28,524 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:28,524 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:28,524 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:28,524 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:28,524 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:28,524 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:28,524 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:28,524 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:28,524 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:28,524 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:28,524 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:28,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60828 deadline: 1680987508523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,524 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:28,524 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:28,524 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:28,525 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,525 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:28,525 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:28,526 DEBUG [Thread-2745] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18205 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be., hostname=jenkins-hbase20.apache.org,41091,1680987313601, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 20:57:28,527 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:28,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:28,527 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:28,527 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,527 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:28,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:28,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:28,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:28,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@176ec6ec[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:28,528 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:28,528 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,528 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,528 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,528 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:28,528 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:28,529 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:28,529 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:28,529 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,529 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:28,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:28,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:28,529 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:28,529 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:28,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:28,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:28,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:28,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:28,531 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:28,531 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:28,531 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:28,531 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:28,531 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:28,534 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,534 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:28,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,534 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:28,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:28,534 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:28,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:28,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:28,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,535 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,537 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:28,537 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:28,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:28,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:28,537 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,537 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:28,537 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:28,537 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:28,537 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:28,537 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:28,537 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,537 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,537 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,537 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,537 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:28,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:28,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:28,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:28,537 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:28,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:28,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987508611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:28,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987508611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,617 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:28,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987508614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,637 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:28,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:28,639 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:28,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:28,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:28,640 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:28,640 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:28,640 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:28,640 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:28,641 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,641 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:28,641 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:28,641 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:28,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:28,642 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,642 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:28,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:28,642 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:28,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:28,642 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:28,642 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:28,642 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:28,643 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:28,643 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:28,643 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:28,643 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:28,643 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:28,643 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:28,643 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:28,643 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:28,643 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:28,643 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:28,643 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:28,643 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,643 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:28,643 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:28,644 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,644 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:28,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:28,644 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,644 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:28,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:28,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:28,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:28,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:28,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@322571ba[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:28,645 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:28,645 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,646 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,646 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:28,646 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:28,646 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:28,647 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,647 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:28,648 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,648 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:28,648 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:28,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:28,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:28,648 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:28,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:28,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:28,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:28,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:28,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:28,650 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:28,650 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:28,650 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:28,650 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:28,650 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:28,650 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,650 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:28,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:28,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,650 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:28,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:28,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:28,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:28,651 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:28,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,652 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:28,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,654 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,654 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:28,654 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:28,654 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:28,654 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:28,654 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:28,654 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:28,654 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:28,654 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:28,654 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,654 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:28,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:28,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:28,654 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,654 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,654 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:28,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:28,681 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/eafed04e10aa452297d59d5ad49b91d9 2023-04-08 20:57:28,754 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:28,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:28,756 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:28,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:28,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:28,757 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:28,757 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:28,757 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:28,757 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:28,757 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,757 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:28,758 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,758 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:28,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:28,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:28,758 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,758 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:28,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:28,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:28,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:28,759 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:28,759 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:28,759 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:28,759 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:28,759 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:28,759 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:28,759 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:28,759 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:28,759 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:28,760 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:28,760 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:28,760 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:28,760 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:28,760 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:28,760 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,760 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:28,760 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:28,761 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,761 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:28,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:28,761 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,761 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:28,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:28,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:28,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:28,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:28,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b71b4ad[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:28,762 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:28,762 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,762 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,762 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:28,762 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,763 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:28,763 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:28,763 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:28,763 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,763 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:28,763 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:28,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:28,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:28,763 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:28,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:28,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:28,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:28,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:28,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:28,765 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:28,765 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:28,765 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:28,765 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:28,765 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:28,766 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,766 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,766 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:28,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:28,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:28,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:28,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:28,766 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:28,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:28,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,767 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:28,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,768 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:28,768 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,769 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:28,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:28,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:28,769 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:28,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:28,769 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:28,769 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:28,769 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,769 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:28,769 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:28,769 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:28,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:28,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:28,769 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:28,769 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,769 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,769 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,869 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:28,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:28,871 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:28,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:28,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:28,872 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:28,872 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:28,872 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:28,872 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:28,872 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,872 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:28,872 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,872 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:28,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:28,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:28,873 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,873 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:28,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:28,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:28,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:28,873 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:28,873 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:28,873 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:28,873 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:28,874 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:28,874 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:28,874 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:28,874 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:28,874 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:28,874 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:28,874 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:28,874 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:28,874 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:28,874 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:28,874 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,874 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:28,874 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:28,875 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,875 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:28,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:28,875 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,875 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:28,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:28,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:28,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:28,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:28,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b533598[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:28,876 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:28,876 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,876 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,877 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:28,877 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:28,877 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:28,877 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,877 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:28,877 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,877 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:28,877 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:28,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:28,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:28,877 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:28,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:28,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:28,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:28,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:28,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:28,879 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:28,879 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:28,879 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:28,879 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:28,879 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:28,880 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,881 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:28,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:28,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,881 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:28,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:28,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:28,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:28,881 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:28,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,882 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:28,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,890 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,890 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:28,890 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:28,890 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:28,890 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,890 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:28,890 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:28,890 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:28,890 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:28,890 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:28,890 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:28,890 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:28,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:28,890 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,890 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:28,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:28,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:28,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987508916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,921 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:28,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987508917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,921 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:28,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987508918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,990 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:28,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:28,992 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:28,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:28,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:28,993 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:28,993 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:28,993 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:28,993 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:28,993 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,993 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:28,994 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:28,994 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:28,994 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,994 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:28,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:28,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:28,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:28,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:28,995 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:28,995 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:28,995 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:28,995 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:28,995 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:28,995 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:28,995 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:28,995 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:28,995 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:28,995 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:28,995 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:28,995 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:28,995 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:28,995 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,996 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:28,996 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:28,996 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,996 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:28,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:28,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:28,997 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,997 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:28,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:28,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:28,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:28,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:28,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a6bd5b8[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:28,998 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:28,998 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,998 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,998 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:28,998 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:28,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:28,999 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:28,999 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:28,999 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:29,000 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:29,000 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:29,000 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:29,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:29,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:29,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:29,000 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:29,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:29,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:29,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:29,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:29,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:29,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:29,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:29,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:29,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:29,002 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:29,002 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:29,002 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:29,002 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:29,002 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:29,003 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:29,003 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:29,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:29,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:29,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:29,003 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:29,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:29,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:29,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:29,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:29,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:29,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:29,004 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:29,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:29,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:29,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:29,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:29,005 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:29,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:29,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:29,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:29,006 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:29,006 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:29,006 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:29,006 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:29,006 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:29,006 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:29,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:29,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:29,006 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:29,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:29,006 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:29,006 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:29,006 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:29,006 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:29,006 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:29,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:29,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:29,006 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:29,006 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:29,091 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/9a39701fee534b6c8681337dad8fc3f2 2023-04-08 20:57:29,098 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/07bff1c521bf4be5bc904b8d47d382e6 2023-04-08 20:57:29,101 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/eafed04e10aa452297d59d5ad49b91d9 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/eafed04e10aa452297d59d5ad49b91d9 2023-04-08 20:57:29,104 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/eafed04e10aa452297d59d5ad49b91d9, entries=200, sequenceid=395, filesize=14.3 K 2023-04-08 20:57:29,104 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/9a39701fee534b6c8681337dad8fc3f2 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/9a39701fee534b6c8681337dad8fc3f2 2023-04-08 20:57:29,107 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:29,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:29,107 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/9a39701fee534b6c8681337dad8fc3f2, entries=150, sequenceid=395, filesize=11.9 K 2023-04-08 20:57:29,108 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/07bff1c521bf4be5bc904b8d47d382e6 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/07bff1c521bf4be5bc904b8d47d382e6 2023-04-08 20:57:29,108 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:29,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:29,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:29,109 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:29,109 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:29,109 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:29,109 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:29,110 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:29,110 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:29,110 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:29,110 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:29,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:29,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:29,111 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/07bff1c521bf4be5bc904b8d47d382e6, entries=150, sequenceid=395, filesize=11.9 K 2023-04-08 20:57:29,111 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:29,111 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:29,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:29,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:29,111 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 0013c9607e8a284ca05e07f23e7866be in 837ms, sequenceid=395, compaction requested=true 2023-04-08 20:57:29,111 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:29,111 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:57:29,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:29,111 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:57:29,111 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:57:29,111 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:57:29,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:29,111 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:57:29,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:29,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:29,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:29,112 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:29,112 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:29,112 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:29,112 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:29,112 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:29,112 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:29,112 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:57:29,112 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:29,112 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39646 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:57:29,112 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 0013c9607e8a284ca05e07f23e7866be/B is initiating minor compaction (all files) 2023-04-08 20:57:29,112 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): 0013c9607e8a284ca05e07f23e7866be/A is initiating minor compaction (all files) 2023-04-08 20:57:29,112 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0013c9607e8a284ca05e07f23e7866be 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-08 20:57:29,112 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0013c9607e8a284ca05e07f23e7866be/A in TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:29,112 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0013c9607e8a284ca05e07f23e7866be/B in TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:29,112 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/4e752c66f00848febab4caec61af74e6, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/ddbc3fad0d5940c5918728cd0f47bc5d, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/eafed04e10aa452297d59d5ad49b91d9] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp, totalSize=38.7 K 2023-04-08 20:57:29,112 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/63f5fa9988714ead91d61e2acb6f0d43, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/794e8162957940acbb9b658a336a85fb, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/9a39701fee534b6c8681337dad8fc3f2] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp, totalSize=36.3 K 2023-04-08 20:57:29,113 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=A 2023-04-08 20:57:29,113 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:29,113 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=B 2023-04-08 20:57:29,113 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 4e752c66f00848febab4caec61af74e6, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1680987446162 2023-04-08 20:57:29,113 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:29,113 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=C 2023-04-08 20:57:29,113 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:29,113 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 63f5fa9988714ead91d61e2acb6f0d43, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1680987446162 2023-04-08 20:57:29,113 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting ddbc3fad0d5940c5918728cd0f47bc5d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1680987446804 2023-04-08 20:57:29,113 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 794e8162957940acbb9b658a336a85fb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1680987446804 2023-04-08 20:57:29,113 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting eafed04e10aa452297d59d5ad49b91d9, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1680987447954 2023-04-08 20:57:29,113 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 9a39701fee534b6c8681337dad8fc3f2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1680987447961 2023-04-08 20:57:29,120 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0013c9607e8a284ca05e07f23e7866be#A#compaction#428 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:57:29,120 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 0013c9607e8a284ca05e07f23e7866be#B#compaction#429 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:57:29,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 0013c9607e8a284ca05e07f23e7866be 2023-04-08 20:57:29,425 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:29,459 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:29,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987509454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:29,461 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:29,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987509457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:29,464 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:29,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987509458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:29,519 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=410 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/52eed597cb4441ababacb0a71f7c156b 2023-04-08 20:57:29,527 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/5d670eb8fb41451982b32175ea46eb11 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/5d670eb8fb41451982b32175ea46eb11 2023-04-08 20:57:29,527 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=410 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/a178750f2f1b4a999ffdae2fa667985c 2023-04-08 20:57:29,533 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0013c9607e8a284ca05e07f23e7866be/B of 0013c9607e8a284ca05e07f23e7866be into 5d670eb8fb41451982b32175ea46eb11(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:57:29,533 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:29,533 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be., storeName=0013c9607e8a284ca05e07f23e7866be/B, priority=13, startTime=1680987449111; duration=0sec 2023-04-08 20:57:29,533 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:57:29,533 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:57:29,534 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/53e8bb0c1cb147cb840bd87375c155da as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/53e8bb0c1cb147cb840bd87375c155da 2023-04-08 20:57:29,535 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:57:29,536 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 0013c9607e8a284ca05e07f23e7866be/C is initiating minor compaction (all files) 2023-04-08 20:57:29,536 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0013c9607e8a284ca05e07f23e7866be/C in TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:29,536 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/fa0f6758a1b04c81bd34d0248d6d6857, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/9a88b91b66a642d594a857a93cd130f6, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/07bff1c521bf4be5bc904b8d47d382e6] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp, totalSize=36.3 K 2023-04-08 20:57:29,536 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting fa0f6758a1b04c81bd34d0248d6d6857, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1680987446162 2023-04-08 20:57:29,536 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 9a88b91b66a642d594a857a93cd130f6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1680987446804 2023-04-08 20:57:29,537 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 07bff1c521bf4be5bc904b8d47d382e6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1680987447961 2023-04-08 20:57:29,539 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0013c9607e8a284ca05e07f23e7866be/A of 0013c9607e8a284ca05e07f23e7866be into 53e8bb0c1cb147cb840bd87375c155da(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:57:29,539 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:29,539 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be., storeName=0013c9607e8a284ca05e07f23e7866be/A, priority=13, startTime=1680987449111; duration=0sec 2023-04-08 20:57:29,539 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:57:29,541 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 0013c9607e8a284ca05e07f23e7866be#C#compaction#432 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:57:29,551 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/25dd96d0437b499da1b051936d367822 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/25dd96d0437b499da1b051936d367822 2023-04-08 20:57:29,555 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0013c9607e8a284ca05e07f23e7866be/C of 0013c9607e8a284ca05e07f23e7866be into 25dd96d0437b499da1b051936d367822(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:57:29,555 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:29,555 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be., storeName=0013c9607e8a284ca05e07f23e7866be/C, priority=13, startTime=1680987449111; duration=0sec 2023-04-08 20:57:29,556 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:57:29,561 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:29,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987509560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:29,565 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:29,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987509561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:29,568 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:29,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987509565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:29,764 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:29,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987509762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:29,768 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:29,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987509765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:29,770 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:29,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987509768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:29,948 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=410 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/4a1ed2c2d3e94358a8f372303b798928 2023-04-08 20:57:29,951 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/52eed597cb4441ababacb0a71f7c156b as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/52eed597cb4441ababacb0a71f7c156b 2023-04-08 20:57:29,954 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/52eed597cb4441ababacb0a71f7c156b, entries=150, sequenceid=410, filesize=11.9 K 2023-04-08 20:57:29,954 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/a178750f2f1b4a999ffdae2fa667985c as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/a178750f2f1b4a999ffdae2fa667985c 2023-04-08 20:57:29,957 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/a178750f2f1b4a999ffdae2fa667985c, entries=150, sequenceid=410, filesize=11.9 K 2023-04-08 20:57:29,957 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/4a1ed2c2d3e94358a8f372303b798928 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/4a1ed2c2d3e94358a8f372303b798928 2023-04-08 20:57:29,960 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/4a1ed2c2d3e94358a8f372303b798928, entries=150, sequenceid=410, filesize=11.9 K 2023-04-08 20:57:29,961 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 0013c9607e8a284ca05e07f23e7866be in 849ms, sequenceid=410, compaction requested=false 2023-04-08 20:57:29,961 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:29,961 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:29,961 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:29,961 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:29,961 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:29,961 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:29,961 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:29,962 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:29,962 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:29,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:29,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:29,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:29,963 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:29,963 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:29,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:29,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:29,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:29,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:29,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:29,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:29,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3dacfee[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:29,964 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:29,964 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:29,964 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:29,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:29,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:29,964 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:29,964 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:29,964 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:29,965 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:29,965 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:29,965 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:29,965 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:29,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:29,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:29,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:29,965 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:29,965 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:29,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:29,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:29,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:29,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:29,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:29,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:29,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:29,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:29,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:29,967 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:29,967 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:29,967 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:29,967 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:29,967 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:29,968 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:29,968 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:29,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:29,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:29,968 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:29,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:29,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:29,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:29,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:29,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:29,968 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:29,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:29,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:29,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:29,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:29,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:29,969 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:29,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:29,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:29,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:29,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:29,971 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:29,971 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:29,971 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:29,971 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:29,971 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:29,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:29,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:29,971 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:29,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:29,971 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:29,971 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:29,971 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:29,971 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:29,971 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:29,971 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:29,971 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:29,971 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:29,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:29,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:30,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 0013c9607e8a284ca05e07f23e7866be 2023-04-08 20:57:30,068 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0013c9607e8a284ca05e07f23e7866be 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-08 20:57:30,068 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=A 2023-04-08 20:57:30,068 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:30,068 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=B 2023-04-08 20:57:30,069 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:30,069 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=C 2023-04-08 20:57:30,069 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:30,071 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:30,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:30,074 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:30,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:30,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:30,075 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:30,075 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:30,075 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:30,075 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:30,076 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:30,076 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:30,076 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:30,076 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:30,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:30,077 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,077 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:30,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:30,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:30,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:30,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:30,077 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:30,077 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:30,077 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:30,077 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:30,078 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:30,078 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:30,078 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:30,078 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:30,078 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:30,078 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:30,078 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:30,078 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:30,078 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:30,078 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:30,078 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,078 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:30,078 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:30,080 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,080 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:30,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:30,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:30,080 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:30,080 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:30,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:30,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:30,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:30,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:30,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:30,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35b8d7e4[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:30,081 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:30,081 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:30,081 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:30,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:30,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:30,081 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:30,081 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:30,081 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,081 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:30,081 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:30,082 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:30,082 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:30,082 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,082 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:30,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:30,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:30,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:30,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:30,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:30,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:30,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:30,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:30,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:30,084 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:30,084 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:30,084 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:30,084 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:30,084 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:30,085 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:30,085 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:30,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:30,085 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:30,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:30,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:30,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:30,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:30,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:30,086 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:30,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:30,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:30,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:30,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:30,086 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:30,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:30,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:30,087 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,087 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:30,087 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:30,087 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:30,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:30,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:30,087 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:30,087 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:30,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:30,088 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:30,088 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:30,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:30,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:30,088 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,088 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:30,088 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:30,088 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:30,088 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:30,088 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:30,096 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:30,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987510090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:30,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987510096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,100 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:30,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987510096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,188 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:30,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:30,190 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:30,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:30,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:30,190 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:30,191 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:30,191 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:30,191 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:30,191 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:30,191 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:30,192 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,192 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:30,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:30,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:30,193 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,193 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:30,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:30,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:30,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:30,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:30,193 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:30,193 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:30,193 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:30,193 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:30,193 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:30,193 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:30,194 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:30,194 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:30,194 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:30,194 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. started... 2023-04-08 20:57:30,194 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. as already flushing 2023-04-08 20:57:30,194 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:30,194 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:30,194 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:30,194 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,194 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:30,194 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:30,195 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,195 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:30,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:30,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:30,196 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:30,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:30,196 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:30,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:30,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:30,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:30,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:30,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5725db3e[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:30,198 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:30,198 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:30,199 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:30,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987510197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,199 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:30,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:30,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:30,199 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:30,199 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:30,199 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:30,199 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,199 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:30,200 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,200 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:30,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:30,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:30,200 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:30,200 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:30,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:30,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:30,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:30,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:30,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:30,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:30,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:30,202 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:30,202 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:30,202 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:30,202 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:30,202 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:30,202 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:30,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987510198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,203 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:30,203 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:30,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:30,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:30,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:30,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:30,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:30,203 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:30,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:30,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:30,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:30,204 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:30,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:30,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:30,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:30,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,204 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:30,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:30,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,205 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:30,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987510201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,206 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:30,206 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:30,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:30,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:30,206 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,206 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:30,206 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:30,206 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:30,206 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:30,206 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:30,206 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,206 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:30,206 DEBUG [Thread-2753] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:30,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:30,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:30,206 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:30,206 DEBUG [Thread-2753] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:30,206 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:30,231 DEBUG [Thread-2754] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c9c1feb to 127.0.0.1:52440 2023-04-08 20:57:30,231 DEBUG [Thread-2754] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:57:30,232 DEBUG [Thread-2756] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x22bf9ddd to 127.0.0.1:52440 2023-04-08 20:57:30,232 DEBUG [Thread-2756] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:57:30,234 DEBUG [Thread-2762] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7bf0ef8b to 127.0.0.1:52440 2023-04-08 20:57:30,235 DEBUG [Thread-2762] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:57:30,239 DEBUG [Thread-2758] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34c4ae9a to 127.0.0.1:52440 2023-04-08 20:57:30,239 DEBUG [Thread-2758] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:57:30,240 DEBUG [Thread-2760] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6187ef67 to 127.0.0.1:52440 2023-04-08 20:57:30,240 DEBUG [Thread-2760] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:57:30,307 DEBUG [Thread-2753] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:30,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:30,403 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:30,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987510402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:30,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987510406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,408 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:30,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987510407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,475 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=431 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/580ed882dfdb421eb7aa8de7ea709483 2023-04-08 20:57:30,707 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:30,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987510707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,710 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:30,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987510710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:30,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987510711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:30,891 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=431 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/999aef045b3f437b9ee9dbe4b8763d00 2023-04-08 20:57:31,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:31,214 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:31,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60788 deadline: 1680987511214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:31,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60798 deadline: 1680987511214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:31,216 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:31,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60824 deadline: 1680987511215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:31,309 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=431 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/a4c86fe1350f48baa6e45b5d1bca8985 2023-04-08 20:57:31,316 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/580ed882dfdb421eb7aa8de7ea709483 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/580ed882dfdb421eb7aa8de7ea709483 2023-04-08 20:57:31,323 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/580ed882dfdb421eb7aa8de7ea709483, entries=200, sequenceid=431, filesize=14.3 K 2023-04-08 20:57:31,324 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/999aef045b3f437b9ee9dbe4b8763d00 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/999aef045b3f437b9ee9dbe4b8763d00 2023-04-08 20:57:31,330 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/999aef045b3f437b9ee9dbe4b8763d00, entries=150, sequenceid=431, filesize=11.9 K 2023-04-08 20:57:31,331 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/a4c86fe1350f48baa6e45b5d1bca8985 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/a4c86fe1350f48baa6e45b5d1bca8985 2023-04-08 20:57:31,336 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/a4c86fe1350f48baa6e45b5d1bca8985, entries=150, sequenceid=431, filesize=11.9 K 2023-04-08 20:57:31,336 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 0013c9607e8a284ca05e07f23e7866be in 1268ms, sequenceid=431, compaction requested=true 2023-04-08 20:57:31,337 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:31,337 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:57:31,337 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:57:31,337 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:57:31,337 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:57:31,337 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:57:31,338 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39748 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:57:31,338 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): 0013c9607e8a284ca05e07f23e7866be/A is initiating minor compaction (all files) 2023-04-08 20:57:31,338 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:57:31,338 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0013c9607e8a284ca05e07f23e7866be/A in TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:31,338 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): 0013c9607e8a284ca05e07f23e7866be/B is initiating minor compaction (all files) 2023-04-08 20:57:31,338 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/53e8bb0c1cb147cb840bd87375c155da, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/52eed597cb4441ababacb0a71f7c156b, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/580ed882dfdb421eb7aa8de7ea709483] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp, totalSize=38.8 K 2023-04-08 20:57:31,338 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0013c9607e8a284ca05e07f23e7866be/B in TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:31,338 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/5d670eb8fb41451982b32175ea46eb11, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/a178750f2f1b4a999ffdae2fa667985c, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/999aef045b3f437b9ee9dbe4b8763d00] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp, totalSize=36.4 K 2023-04-08 20:57:31,339 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 53e8bb0c1cb147cb840bd87375c155da, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1680987447961 2023-04-08 20:57:31,339 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 5d670eb8fb41451982b32175ea46eb11, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1680987447961 2023-04-08 20:57:31,339 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 52eed597cb4441ababacb0a71f7c156b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1680987448282 2023-04-08 20:57:31,339 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting a178750f2f1b4a999ffdae2fa667985c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1680987448282 2023-04-08 20:57:31,340 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 580ed882dfdb421eb7aa8de7ea709483, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1680987449449 2023-04-08 20:57:31,340 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 999aef045b3f437b9ee9dbe4b8763d00, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1680987449456 2023-04-08 20:57:31,348 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): 0013c9607e8a284ca05e07f23e7866be#B#compaction#437 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:57:31,348 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0013c9607e8a284ca05e07f23e7866be#A#compaction#436 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:57:31,764 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/cf650818d08b445fa6d75377ff51a8c8 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/cf650818d08b445fa6d75377ff51a8c8 2023-04-08 20:57:31,765 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/bd892a777e01466b86d290595acfbb1c as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/bd892a777e01466b86d290595acfbb1c 2023-04-08 20:57:31,769 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0013c9607e8a284ca05e07f23e7866be/A of 0013c9607e8a284ca05e07f23e7866be into bd892a777e01466b86d290595acfbb1c(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:57:31,769 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0013c9607e8a284ca05e07f23e7866be/B of 0013c9607e8a284ca05e07f23e7866be into cf650818d08b445fa6d75377ff51a8c8(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:57:31,769 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:31,769 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:31,769 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be., storeName=0013c9607e8a284ca05e07f23e7866be/A, priority=13, startTime=1680987451337; duration=0sec 2023-04-08 20:57:31,770 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be., storeName=0013c9607e8a284ca05e07f23e7866be/B, priority=13, startTime=1680987451337; duration=0sec 2023-04-08 20:57:31,770 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:57:31,770 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:57:31,770 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 20:57:31,771 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 20:57:31,771 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): 0013c9607e8a284ca05e07f23e7866be/C is initiating minor compaction (all files) 2023-04-08 20:57:31,771 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0013c9607e8a284ca05e07f23e7866be/C in TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:31,771 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/25dd96d0437b499da1b051936d367822, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/4a1ed2c2d3e94358a8f372303b798928, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/a4c86fe1350f48baa6e45b5d1bca8985] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp, totalSize=36.4 K 2023-04-08 20:57:31,772 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 25dd96d0437b499da1b051936d367822, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1680987447961 2023-04-08 20:57:31,772 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 4a1ed2c2d3e94358a8f372303b798928, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1680987448282 2023-04-08 20:57:31,772 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting a4c86fe1350f48baa6e45b5d1bca8985, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1680987449456 2023-04-08 20:57:31,779 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0013c9607e8a284ca05e07f23e7866be#C#compaction#438 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:57:32,194 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/d28d38d440b0460285a7fd2f29b9eb58 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/d28d38d440b0460285a7fd2f29b9eb58 2023-04-08 20:57:32,202 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0013c9607e8a284ca05e07f23e7866be/C of 0013c9607e8a284ca05e07f23e7866be into d28d38d440b0460285a7fd2f29b9eb58(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:57:32,202 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:32,202 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be., storeName=0013c9607e8a284ca05e07f23e7866be/C, priority=13, startTime=1680987451337; duration=0sec 2023-04-08 20:57:32,202 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:57:32,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on 0013c9607e8a284ca05e07f23e7866be 2023-04-08 20:57:32,220 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0013c9607e8a284ca05e07f23e7866be 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-08 20:57:32,220 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=A 2023-04-08 20:57:32,220 DEBUG [Thread-2747] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4c4d3f2e to 127.0.0.1:52440 2023-04-08 20:57:32,220 DEBUG [Thread-2747] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:57:32,220 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:32,220 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=B 2023-04-08 20:57:32,220 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:32,221 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=C 2023-04-08 20:57:32,221 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:32,226 DEBUG [Thread-2743] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x50e89dd2 to 127.0.0.1:52440 2023-04-08 20:57:32,226 DEBUG [Thread-2743] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:57:32,228 DEBUG [Thread-2749] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x47d58a9b to 127.0.0.1:52440 2023-04-08 20:57:32,228 DEBUG [Thread-2749] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:57:32,631 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=449 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/d682a2387afb4bf294df86fa95183940 2023-04-08 20:57:33,042 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=449 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/67fc7cb00efc49c98223b13177a8dbdc 2023-04-08 20:57:33,459 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=449 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/8440c11a512d4b2b9e9585526bd374e0 2023-04-08 20:57:33,466 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/d682a2387afb4bf294df86fa95183940 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/d682a2387afb4bf294df86fa95183940 2023-04-08 20:57:33,471 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/d682a2387afb4bf294df86fa95183940, entries=150, sequenceid=449, filesize=11.9 K 2023-04-08 20:57:33,472 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/67fc7cb00efc49c98223b13177a8dbdc as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/67fc7cb00efc49c98223b13177a8dbdc 2023-04-08 20:57:33,477 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/67fc7cb00efc49c98223b13177a8dbdc, entries=150, sequenceid=449, filesize=11.9 K 2023-04-08 20:57:33,477 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/8440c11a512d4b2b9e9585526bd374e0 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/8440c11a512d4b2b9e9585526bd374e0 2023-04-08 20:57:33,481 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/8440c11a512d4b2b9e9585526bd374e0, entries=150, sequenceid=449, filesize=11.9 K 2023-04-08 20:57:33,482 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=13.42 KB/13740 for 0013c9607e8a284ca05e07f23e7866be in 1262ms, sequenceid=449, compaction requested=false 2023-04-08 20:57:33,482 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:38,512 DEBUG [Thread-2751] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3af76165 to 127.0.0.1:52440 2023-04-08 20:57:38,512 DEBUG [Thread-2751] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:57:38,556 DEBUG [Thread-2745] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x545adde0 to 127.0.0.1:52440 2023-04-08 20:57:38,556 DEBUG [Thread-2745] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:57:38,556 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-08 20:57:38,557 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(394): wrote 107 2023-04-08 20:57:38,557 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(394): wrote 17 2023-04-08 20:57:38,557 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(394): wrote 112 2023-04-08 20:57:38,557 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(394): wrote 109 2023-04-08 20:57:38,557 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(394): wrote 17 2023-04-08 20:57:38,557 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-08 20:57:38,557 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-08 20:57:38,558 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(402): scanned 2261 2023-04-08 20:57:38,558 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(403): verified 6783 rows 2023-04-08 20:57:38,558 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(402): scanned 2255 2023-04-08 20:57:38,558 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(403): verified 6765 rows 2023-04-08 20:57:38,558 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(402): scanned 2266 2023-04-08 20:57:38,558 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(403): verified 6795 rows 2023-04-08 20:57:38,558 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(402): scanned 2250 2023-04-08 20:57:38,558 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(403): verified 6750 rows 2023-04-08 20:57:38,558 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(402): scanned 2253 2023-04-08 20:57:38,558 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(403): verified 6759 rows 2023-04-08 20:57:38,559 INFO [Listener at localhost.localdomain/41823] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-08 20:57:38,559 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7cb49278 to 127.0.0.1:52440 2023-04-08 20:57:38,559 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:57:38,561 INFO [Listener at localhost.localdomain/41823] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-08 20:57:38,562 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-08 20:57:38,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-08 20:57:38,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-08 20:57:38,566 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680987458566"}]},"ts":"1680987458566"} 2023-04-08 20:57:38,567 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-08 20:57:38,572 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-08 20:57:38,573 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0013c9607e8a284ca05e07f23e7866be, UNASSIGN}] 2023-04-08 20:57:38,573 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0013c9607e8a284ca05e07f23e7866be, UNASSIGN 2023-04-08 20:57:38,573 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=0013c9607e8a284ca05e07f23e7866be, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:38,574 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987458573"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680987458573"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680987458573"}]},"ts":"1680987458573"} 2023-04-08 20:57:38,574 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601}] 2023-04-08 20:57:38,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-08 20:57:38,726 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 0013c9607e8a284ca05e07f23e7866be 2023-04-08 20:57:38,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 0013c9607e8a284ca05e07f23e7866be, disabling compactions & flushes 2023-04-08 20:57:38,727 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:38,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:38,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. after waiting 0 ms 2023-04-08 20:57:38,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:38,727 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 0013c9607e8a284ca05e07f23e7866be 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-08 20:57:38,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=A 2023-04-08 20:57:38,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:38,729 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=B 2023-04-08 20:57:38,729 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:38,729 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0013c9607e8a284ca05e07f23e7866be, store=C 2023-04-08 20:57:38,729 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:38,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-08 20:57:39,141 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/4728209720e148f6b35e36e25d8cd520 2023-04-08 20:57:39,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-08 20:57:39,558 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/58f1519d7b30471ca7bbe5cbdb20c6bf 2023-04-08 20:57:39,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-08 20:57:39,970 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/5285f882a3044ef3b714e067aa671487 2023-04-08 20:57:39,978 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/A/4728209720e148f6b35e36e25d8cd520 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/4728209720e148f6b35e36e25d8cd520 2023-04-08 20:57:39,986 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/4728209720e148f6b35e36e25d8cd520, entries=150, sequenceid=456, filesize=11.9 K 2023-04-08 20:57:39,988 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/B/58f1519d7b30471ca7bbe5cbdb20c6bf as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/58f1519d7b30471ca7bbe5cbdb20c6bf 2023-04-08 20:57:39,993 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/58f1519d7b30471ca7bbe5cbdb20c6bf, entries=150, sequenceid=456, filesize=11.9 K 2023-04-08 20:57:39,994 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/.tmp/C/5285f882a3044ef3b714e067aa671487 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/5285f882a3044ef3b714e067aa671487 2023-04-08 20:57:39,999 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/5285f882a3044ef3b714e067aa671487, entries=150, sequenceid=456, filesize=11.9 K 2023-04-08 20:57:40,000 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 0013c9607e8a284ca05e07f23e7866be in 1273ms, sequenceid=456, compaction requested=true 2023-04-08 20:57:40,023 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/8e508f63691e4e558db0f7a83d2f832a, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/d5330807be7240579d405c330845786a, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/c9c359ce6ff94d6e9c28409d50e00739, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/ccd459f4d4cd48c3b786612c668a0d4e, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/6d6776d152cb4a4798d78ae194ccb82c, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/4f092bb0d2d24a07929c0cddd75cce51, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/e6405254d9a042cb9e572a4235d7dbce, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/5a838769bf93483da8947c8f75fd342c, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/ec5c3899e7d745f4b2aa5fd197533ee1, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/321999a731bb494fa12c3c5d1f8d8db8, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/1b02e15a972e44fc8b0d3d8314956966, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/91b6dfc00d844addba3538824a795d23, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/fe79324a6a174137a15accb76b5080cc, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/890f2082a0534325a50128ede792bf7b, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/5baa9793369640548636e2354be557f0, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/af720d4c005342d39d23737ddaa3a171, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/08772e680fc848f29593c5e8bb0b9458, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/55db870507b64d91b43d16d3530500d7, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/4e752c66f00848febab4caec61af74e6, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/9381f4a531b34b4c952cae624f73a919, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/ddbc3fad0d5940c5918728cd0f47bc5d, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/eafed04e10aa452297d59d5ad49b91d9, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/53e8bb0c1cb147cb840bd87375c155da, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/52eed597cb4441ababacb0a71f7c156b, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/580ed882dfdb421eb7aa8de7ea709483] to archive 2023-04-08 20:57:40,023 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 20:57:40,025 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/8e508f63691e4e558db0f7a83d2f832a to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/8e508f63691e4e558db0f7a83d2f832a 2023-04-08 20:57:40,026 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/d5330807be7240579d405c330845786a to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/d5330807be7240579d405c330845786a 2023-04-08 20:57:40,028 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/c9c359ce6ff94d6e9c28409d50e00739 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/c9c359ce6ff94d6e9c28409d50e00739 2023-04-08 20:57:40,028 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/ccd459f4d4cd48c3b786612c668a0d4e to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/ccd459f4d4cd48c3b786612c668a0d4e 2023-04-08 20:57:40,029 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/6d6776d152cb4a4798d78ae194ccb82c to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/6d6776d152cb4a4798d78ae194ccb82c 2023-04-08 20:57:40,030 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/4f092bb0d2d24a07929c0cddd75cce51 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/4f092bb0d2d24a07929c0cddd75cce51 2023-04-08 20:57:40,031 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/e6405254d9a042cb9e572a4235d7dbce to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/e6405254d9a042cb9e572a4235d7dbce 2023-04-08 20:57:40,032 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/5a838769bf93483da8947c8f75fd342c to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/5a838769bf93483da8947c8f75fd342c 2023-04-08 20:57:40,032 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/ec5c3899e7d745f4b2aa5fd197533ee1 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/ec5c3899e7d745f4b2aa5fd197533ee1 2023-04-08 20:57:40,033 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/321999a731bb494fa12c3c5d1f8d8db8 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/321999a731bb494fa12c3c5d1f8d8db8 2023-04-08 20:57:40,034 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/1b02e15a972e44fc8b0d3d8314956966 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/1b02e15a972e44fc8b0d3d8314956966 2023-04-08 20:57:40,035 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/91b6dfc00d844addba3538824a795d23 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/91b6dfc00d844addba3538824a795d23 2023-04-08 20:57:40,036 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/fe79324a6a174137a15accb76b5080cc to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/fe79324a6a174137a15accb76b5080cc 2023-04-08 20:57:40,037 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/890f2082a0534325a50128ede792bf7b to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/890f2082a0534325a50128ede792bf7b 2023-04-08 20:57:40,038 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/5baa9793369640548636e2354be557f0 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/5baa9793369640548636e2354be557f0 2023-04-08 20:57:40,039 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/af720d4c005342d39d23737ddaa3a171 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/af720d4c005342d39d23737ddaa3a171 2023-04-08 20:57:40,040 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/08772e680fc848f29593c5e8bb0b9458 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/08772e680fc848f29593c5e8bb0b9458 2023-04-08 20:57:40,041 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/55db870507b64d91b43d16d3530500d7 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/55db870507b64d91b43d16d3530500d7 2023-04-08 20:57:40,042 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/4e752c66f00848febab4caec61af74e6 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/4e752c66f00848febab4caec61af74e6 2023-04-08 20:57:40,043 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/9381f4a531b34b4c952cae624f73a919 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/9381f4a531b34b4c952cae624f73a919 2023-04-08 20:57:40,044 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/ddbc3fad0d5940c5918728cd0f47bc5d to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/ddbc3fad0d5940c5918728cd0f47bc5d 2023-04-08 20:57:40,045 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/eafed04e10aa452297d59d5ad49b91d9 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/eafed04e10aa452297d59d5ad49b91d9 2023-04-08 20:57:40,046 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/53e8bb0c1cb147cb840bd87375c155da to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/53e8bb0c1cb147cb840bd87375c155da 2023-04-08 20:57:40,047 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/52eed597cb4441ababacb0a71f7c156b to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/52eed597cb4441ababacb0a71f7c156b 2023-04-08 20:57:40,048 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/580ed882dfdb421eb7aa8de7ea709483 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/580ed882dfdb421eb7aa8de7ea709483 2023-04-08 20:57:40,051 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/58e8e8f8b93a4a3292a08281b3fdd82a, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/7bd446122bf546c89c0da304c2584963, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/f55aae46c163470898316109daaed75a, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/fc240fb569894058b5a7280102df68a1, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/fd074d2da18640f4b92b22a6612af1f4, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/3d983b9d1b914ea8af998ebe0b19e045, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/7eef716d7df642d3837163baa032ef7e, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/649fbd0af4e447fea3389ca1419c22e9, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/a5aeb67cd58d44829103d32d7d99153f, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/22617e11ffac47bf977e0533d1a35d39, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/496ac6ea10624e4c81265a6a0b80c79a, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/6100dafa9531443b8b59029bbff23424, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/4768bc1a32304346bf21511c21129d1d, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/64aedb613dad451090529a0c6315db60, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/962a58af2aea4dbea5c1efd9fa32412a, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/70d13def61f641bc9ee8df353450dd5c, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/32edda62db114ba48b7da78a99056f3b, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/4f28b4fedb4649c994da79489ecaf893, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/63f5fa9988714ead91d61e2acb6f0d43, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/efd5bcc5cac24c5cb1c7288f7d7b6665, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/794e8162957940acbb9b658a336a85fb, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/5d670eb8fb41451982b32175ea46eb11, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/9a39701fee534b6c8681337dad8fc3f2, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/a178750f2f1b4a999ffdae2fa667985c, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/999aef045b3f437b9ee9dbe4b8763d00] to archive 2023-04-08 20:57:40,052 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 20:57:40,053 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/58e8e8f8b93a4a3292a08281b3fdd82a to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/58e8e8f8b93a4a3292a08281b3fdd82a 2023-04-08 20:57:40,054 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/7bd446122bf546c89c0da304c2584963 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/7bd446122bf546c89c0da304c2584963 2023-04-08 20:57:40,055 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/f55aae46c163470898316109daaed75a to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/f55aae46c163470898316109daaed75a 2023-04-08 20:57:40,056 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/fc240fb569894058b5a7280102df68a1 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/fc240fb569894058b5a7280102df68a1 2023-04-08 20:57:40,057 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/fd074d2da18640f4b92b22a6612af1f4 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/fd074d2da18640f4b92b22a6612af1f4 2023-04-08 20:57:40,058 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/3d983b9d1b914ea8af998ebe0b19e045 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/3d983b9d1b914ea8af998ebe0b19e045 2023-04-08 20:57:40,060 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/7eef716d7df642d3837163baa032ef7e to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/7eef716d7df642d3837163baa032ef7e 2023-04-08 20:57:40,061 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/649fbd0af4e447fea3389ca1419c22e9 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/649fbd0af4e447fea3389ca1419c22e9 2023-04-08 20:57:40,062 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/a5aeb67cd58d44829103d32d7d99153f to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/a5aeb67cd58d44829103d32d7d99153f 2023-04-08 20:57:40,063 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/22617e11ffac47bf977e0533d1a35d39 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/22617e11ffac47bf977e0533d1a35d39 2023-04-08 20:57:40,064 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/496ac6ea10624e4c81265a6a0b80c79a to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/496ac6ea10624e4c81265a6a0b80c79a 2023-04-08 20:57:40,066 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/6100dafa9531443b8b59029bbff23424 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/6100dafa9531443b8b59029bbff23424 2023-04-08 20:57:40,067 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/4768bc1a32304346bf21511c21129d1d to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/4768bc1a32304346bf21511c21129d1d 2023-04-08 20:57:40,068 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/64aedb613dad451090529a0c6315db60 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/64aedb613dad451090529a0c6315db60 2023-04-08 20:57:40,069 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/962a58af2aea4dbea5c1efd9fa32412a to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/962a58af2aea4dbea5c1efd9fa32412a 2023-04-08 20:57:40,070 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/70d13def61f641bc9ee8df353450dd5c to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/70d13def61f641bc9ee8df353450dd5c 2023-04-08 20:57:40,072 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/32edda62db114ba48b7da78a99056f3b to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/32edda62db114ba48b7da78a99056f3b 2023-04-08 20:57:40,073 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/4f28b4fedb4649c994da79489ecaf893 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/4f28b4fedb4649c994da79489ecaf893 2023-04-08 20:57:40,075 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/63f5fa9988714ead91d61e2acb6f0d43 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/63f5fa9988714ead91d61e2acb6f0d43 2023-04-08 20:57:40,076 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/efd5bcc5cac24c5cb1c7288f7d7b6665 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/efd5bcc5cac24c5cb1c7288f7d7b6665 2023-04-08 20:57:40,077 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/794e8162957940acbb9b658a336a85fb to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/794e8162957940acbb9b658a336a85fb 2023-04-08 20:57:40,078 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/5d670eb8fb41451982b32175ea46eb11 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/5d670eb8fb41451982b32175ea46eb11 2023-04-08 20:57:40,080 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/9a39701fee534b6c8681337dad8fc3f2 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/9a39701fee534b6c8681337dad8fc3f2 2023-04-08 20:57:40,081 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/a178750f2f1b4a999ffdae2fa667985c to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/a178750f2f1b4a999ffdae2fa667985c 2023-04-08 20:57:40,082 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/999aef045b3f437b9ee9dbe4b8763d00 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/999aef045b3f437b9ee9dbe4b8763d00 2023-04-08 20:57:40,086 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/8a9d563c538941238460b51ebfafed8f, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/a915bdce531642a894e51bd7bb085b63, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/a5ab98d7fd5b4c6e8568e6c830f74d52, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/3e199f3e6cde4e2f8cacb03766e5a6ed, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/a0a252e001c54dc897ba0db5ae487417, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/fb0c90f524f541d787a6a7e8bf126d13, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/c71a49945aae4a158380a5e26d1e8365, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/90934e93d68743d4b72b4cdeb7046d35, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/ba91a4d6099f4fb78196dbabe70bd690, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/ddfd72e78aff4b7abe3ae11634a808bc, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/ad905d732fcd494981238323edb53537, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/1d2c79ef01d043d0bebf2939ae8fa32f, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/a2028e9fe2d04b1d982332dc365341a6, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/a2cdc2252fc04a4698cb688f5f3713dd, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/644c38b4e2854999975e07e9f21d8705, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/06065dcaecfa490582e70ed79cbf0c66, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/3ab1eca87c064fae872ec6aebc9f76ce, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/2ffaaf12669747d094f66835546a82b0, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/fa0f6758a1b04c81bd34d0248d6d6857, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/e40267049e614350ae8d901d067c1e2b, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/9a88b91b66a642d594a857a93cd130f6, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/25dd96d0437b499da1b051936d367822, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/07bff1c521bf4be5bc904b8d47d382e6, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/4a1ed2c2d3e94358a8f372303b798928, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/a4c86fe1350f48baa6e45b5d1bca8985] to archive 2023-04-08 20:57:40,087 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 20:57:40,088 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/8a9d563c538941238460b51ebfafed8f to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/8a9d563c538941238460b51ebfafed8f 2023-04-08 20:57:40,090 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/a915bdce531642a894e51bd7bb085b63 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/a915bdce531642a894e51bd7bb085b63 2023-04-08 20:57:40,092 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/a5ab98d7fd5b4c6e8568e6c830f74d52 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/a5ab98d7fd5b4c6e8568e6c830f74d52 2023-04-08 20:57:40,093 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/3e199f3e6cde4e2f8cacb03766e5a6ed to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/3e199f3e6cde4e2f8cacb03766e5a6ed 2023-04-08 20:57:40,094 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/a0a252e001c54dc897ba0db5ae487417 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/a0a252e001c54dc897ba0db5ae487417 2023-04-08 20:57:40,095 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/fb0c90f524f541d787a6a7e8bf126d13 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/fb0c90f524f541d787a6a7e8bf126d13 2023-04-08 20:57:40,096 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/c71a49945aae4a158380a5e26d1e8365 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/c71a49945aae4a158380a5e26d1e8365 2023-04-08 20:57:40,096 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/90934e93d68743d4b72b4cdeb7046d35 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/90934e93d68743d4b72b4cdeb7046d35 2023-04-08 20:57:40,097 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/ba91a4d6099f4fb78196dbabe70bd690 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/ba91a4d6099f4fb78196dbabe70bd690 2023-04-08 20:57:40,098 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/ddfd72e78aff4b7abe3ae11634a808bc to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/ddfd72e78aff4b7abe3ae11634a808bc 2023-04-08 20:57:40,099 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/ad905d732fcd494981238323edb53537 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/ad905d732fcd494981238323edb53537 2023-04-08 20:57:40,099 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/1d2c79ef01d043d0bebf2939ae8fa32f to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/1d2c79ef01d043d0bebf2939ae8fa32f 2023-04-08 20:57:40,100 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/a2028e9fe2d04b1d982332dc365341a6 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/a2028e9fe2d04b1d982332dc365341a6 2023-04-08 20:57:40,100 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/a2cdc2252fc04a4698cb688f5f3713dd to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/a2cdc2252fc04a4698cb688f5f3713dd 2023-04-08 20:57:40,101 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/644c38b4e2854999975e07e9f21d8705 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/644c38b4e2854999975e07e9f21d8705 2023-04-08 20:57:40,102 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/06065dcaecfa490582e70ed79cbf0c66 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/06065dcaecfa490582e70ed79cbf0c66 2023-04-08 20:57:40,102 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/3ab1eca87c064fae872ec6aebc9f76ce to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/3ab1eca87c064fae872ec6aebc9f76ce 2023-04-08 20:57:40,103 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/2ffaaf12669747d094f66835546a82b0 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/2ffaaf12669747d094f66835546a82b0 2023-04-08 20:57:40,104 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/fa0f6758a1b04c81bd34d0248d6d6857 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/fa0f6758a1b04c81bd34d0248d6d6857 2023-04-08 20:57:40,104 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/e40267049e614350ae8d901d067c1e2b to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/e40267049e614350ae8d901d067c1e2b 2023-04-08 20:57:40,105 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/9a88b91b66a642d594a857a93cd130f6 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/9a88b91b66a642d594a857a93cd130f6 2023-04-08 20:57:40,106 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/25dd96d0437b499da1b051936d367822 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/25dd96d0437b499da1b051936d367822 2023-04-08 20:57:40,107 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/07bff1c521bf4be5bc904b8d47d382e6 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/07bff1c521bf4be5bc904b8d47d382e6 2023-04-08 20:57:40,108 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/4a1ed2c2d3e94358a8f372303b798928 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/4a1ed2c2d3e94358a8f372303b798928 2023-04-08 20:57:40,109 DEBUG [StoreCloser-TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/a4c86fe1350f48baa6e45b5d1bca8985 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/a4c86fe1350f48baa6e45b5d1bca8985 2023-04-08 20:57:40,114 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/recovered.edits/459.seqid, newMaxSeqId=459, maxSeqId=1 2023-04-08 20:57:40,116 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be. 2023-04-08 20:57:40,116 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 0013c9607e8a284ca05e07f23e7866be: 2023-04-08 20:57:40,118 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 0013c9607e8a284ca05e07f23e7866be 2023-04-08 20:57:40,118 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=0013c9607e8a284ca05e07f23e7866be, regionState=CLOSED 2023-04-08 20:57:40,119 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987460118"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680987460118"}]},"ts":"1680987460118"} 2023-04-08 20:57:40,121 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-08 20:57:40,121 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 0013c9607e8a284ca05e07f23e7866be, server=jenkins-hbase20.apache.org,41091,1680987313601 in 1.5460 sec 2023-04-08 20:57:40,123 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-08 20:57:40,123 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=0013c9607e8a284ca05e07f23e7866be, UNASSIGN in 1.5480 sec 2023-04-08 20:57:40,123 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680987460123"}]},"ts":"1680987460123"} 2023-04-08 20:57:40,124 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-08 20:57:40,125 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-08 20:57:40,126 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5630 sec 2023-04-08 20:57:40,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-08 20:57:40,672 INFO [Listener at localhost.localdomain/41823] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-08 20:57:40,673 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-08 20:57:40,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 20:57:40,676 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 20:57:40,677 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 20:57:40,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-08 20:57:40,683 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be 2023-04-08 20:57:40,687 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A, FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B, FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C, FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/recovered.edits] 2023-04-08 20:57:40,692 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/4728209720e148f6b35e36e25d8cd520 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/4728209720e148f6b35e36e25d8cd520 2023-04-08 20:57:40,695 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/bd892a777e01466b86d290595acfbb1c to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/bd892a777e01466b86d290595acfbb1c 2023-04-08 20:57:40,697 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/d682a2387afb4bf294df86fa95183940 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/A/d682a2387afb4bf294df86fa95183940 2023-04-08 20:57:40,702 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/58f1519d7b30471ca7bbe5cbdb20c6bf to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/58f1519d7b30471ca7bbe5cbdb20c6bf 2023-04-08 20:57:40,704 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/67fc7cb00efc49c98223b13177a8dbdc to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/67fc7cb00efc49c98223b13177a8dbdc 2023-04-08 20:57:40,706 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/cf650818d08b445fa6d75377ff51a8c8 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/B/cf650818d08b445fa6d75377ff51a8c8 2023-04-08 20:57:40,710 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/5285f882a3044ef3b714e067aa671487 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/5285f882a3044ef3b714e067aa671487 2023-04-08 20:57:40,711 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/8440c11a512d4b2b9e9585526bd374e0 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/8440c11a512d4b2b9e9585526bd374e0 2023-04-08 20:57:40,713 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/d28d38d440b0460285a7fd2f29b9eb58 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/C/d28d38d440b0460285a7fd2f29b9eb58 2023-04-08 20:57:40,717 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/recovered.edits/459.seqid to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be/recovered.edits/459.seqid 2023-04-08 20:57:40,718 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/0013c9607e8a284ca05e07f23e7866be 2023-04-08 20:57:40,718 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-08 20:57:40,721 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 20:57:40,722 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-08 20:57:40,724 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-08 20:57:40,725 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 20:57:40,726 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-08 20:57:40,726 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680987460726"}]},"ts":"9223372036854775807"} 2023-04-08 20:57:40,728 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-08 20:57:40,728 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 0013c9607e8a284ca05e07f23e7866be, NAME => 'TestAcidGuarantees,,1680987428013.0013c9607e8a284ca05e07f23e7866be.', STARTKEY => '', ENDKEY => ''}] 2023-04-08 20:57:40,728 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-08 20:57:40,728 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680987460728"}]},"ts":"9223372036854775807"} 2023-04-08 20:57:40,730 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-08 20:57:40,731 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 20:57:40,733 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 59 msec 2023-04-08 20:57:40,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-08 20:57:40,779 INFO [Listener at localhost.localdomain/41823] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-08 20:57:40,796 INFO [Listener at localhost.localdomain/41823] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=225 (was 227), OpenFileDescriptor=544 (was 511) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=303 (was 358), ProcessCount=166 (was 166), AvailableMemoryMB=3592 (was 3510) - AvailableMemoryMB LEAK? - 2023-04-08 20:57:40,809 INFO [Listener at localhost.localdomain/41823] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225, OpenFileDescriptor=544, MaxFileDescriptor=60000, SystemLoadAverage=303, ProcessCount=166, AvailableMemoryMB=3591 2023-04-08 20:57:40,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-08 20:57:40,811 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-08 20:57:40,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-08 20:57:40,813 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-08 20:57:40,813 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-08 20:57:40,814 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-08 20:57:40,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-08 20:57:40,815 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:57:40,815 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac empty. 2023-04-08 20:57:40,815 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:57:40,815 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-08 20:57:40,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-08 20:57:41,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-08 20:57:41,223 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-08 20:57:41,225 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => e986ad248e3bbfc4197e05257d0f5eac, NAME => 'TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp 2023-04-08 20:57:41,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-08 20:57:41,634 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 20:57:41,635 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing e986ad248e3bbfc4197e05257d0f5eac, disabling compactions & flushes 2023-04-08 20:57:41,635 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:41,635 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:41,635 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. after waiting 0 ms 2023-04-08 20:57:41,635 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:41,635 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:41,635 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for e986ad248e3bbfc4197e05257d0f5eac: 2023-04-08 20:57:41,640 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-08 20:57:41,642 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987461642"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680987461642"}]},"ts":"1680987461642"} 2023-04-08 20:57:41,644 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-08 20:57:41,645 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-08 20:57:41,645 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680987461645"}]},"ts":"1680987461645"} 2023-04-08 20:57:41,647 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-08 20:57:41,651 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e986ad248e3bbfc4197e05257d0f5eac, ASSIGN}] 2023-04-08 20:57:41,653 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e986ad248e3bbfc4197e05257d0f5eac, ASSIGN 2023-04-08 20:57:41,654 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=e986ad248e3bbfc4197e05257d0f5eac, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,41091,1680987313601; forceNewPlan=false, retain=false 2023-04-08 20:57:41,805 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=e986ad248e3bbfc4197e05257d0f5eac, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:41,806 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987461805"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680987461805"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680987461805"}]},"ts":"1680987461805"} 2023-04-08 20:57:41,807 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601}] 2023-04-08 20:57:41,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-08 20:57:41,964 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:41,964 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e986ad248e3bbfc4197e05257d0f5eac, NAME => 'TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.', STARTKEY => '', ENDKEY => ''} 2023-04-08 20:57:41,964 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:57:41,964 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 20:57:41,965 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:57:41,965 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:57:41,966 INFO [StoreOpener-e986ad248e3bbfc4197e05257d0f5eac-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:57:41,968 DEBUG [StoreOpener-e986ad248e3bbfc4197e05257d0f5eac-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A 2023-04-08 20:57:41,968 DEBUG [StoreOpener-e986ad248e3bbfc4197e05257d0f5eac-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A 2023-04-08 20:57:41,968 INFO [StoreOpener-e986ad248e3bbfc4197e05257d0f5eac-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 20:57:41,968 INFO [StoreOpener-e986ad248e3bbfc4197e05257d0f5eac-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e986ad248e3bbfc4197e05257d0f5eac columnFamilyName A 2023-04-08 20:57:41,968 INFO [StoreOpener-e986ad248e3bbfc4197e05257d0f5eac-1] regionserver.HStore(310): Store=e986ad248e3bbfc4197e05257d0f5eac/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 20:57:41,969 INFO [StoreOpener-e986ad248e3bbfc4197e05257d0f5eac-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:57:41,970 DEBUG [StoreOpener-e986ad248e3bbfc4197e05257d0f5eac-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B 2023-04-08 20:57:41,970 DEBUG [StoreOpener-e986ad248e3bbfc4197e05257d0f5eac-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B 2023-04-08 20:57:41,970 INFO [StoreOpener-e986ad248e3bbfc4197e05257d0f5eac-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 20:57:41,970 INFO [StoreOpener-e986ad248e3bbfc4197e05257d0f5eac-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e986ad248e3bbfc4197e05257d0f5eac columnFamilyName B 2023-04-08 20:57:41,970 INFO [StoreOpener-e986ad248e3bbfc4197e05257d0f5eac-1] regionserver.HStore(310): Store=e986ad248e3bbfc4197e05257d0f5eac/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 20:57:41,971 INFO [StoreOpener-e986ad248e3bbfc4197e05257d0f5eac-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:57:41,972 DEBUG [StoreOpener-e986ad248e3bbfc4197e05257d0f5eac-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C 2023-04-08 20:57:41,972 DEBUG [StoreOpener-e986ad248e3bbfc4197e05257d0f5eac-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C 2023-04-08 20:57:41,972 INFO [StoreOpener-e986ad248e3bbfc4197e05257d0f5eac-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 20:57:41,972 INFO [StoreOpener-e986ad248e3bbfc4197e05257d0f5eac-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e986ad248e3bbfc4197e05257d0f5eac columnFamilyName C 2023-04-08 20:57:41,973 INFO [StoreOpener-e986ad248e3bbfc4197e05257d0f5eac-1] regionserver.HStore(310): Store=e986ad248e3bbfc4197e05257d0f5eac/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 20:57:41,973 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:41,973 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:57:41,974 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:57:41,976 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-08 20:57:41,977 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:57:41,979 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-08 20:57:41,979 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened e986ad248e3bbfc4197e05257d0f5eac; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66039454, jitterRate=-0.015935450792312622}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-08 20:57:41,979 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for e986ad248e3bbfc4197e05257d0f5eac: 2023-04-08 20:57:41,980 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac., pid=56, masterSystemTime=1680987461959 2023-04-08 20:57:41,981 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:41,981 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:41,981 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=e986ad248e3bbfc4197e05257d0f5eac, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:41,981 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987461981"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680987461981"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680987461981"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680987461981"}]},"ts":"1680987461981"} 2023-04-08 20:57:41,983 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-08 20:57:41,984 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 in 175 msec 2023-04-08 20:57:41,985 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-08 20:57:41,985 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e986ad248e3bbfc4197e05257d0f5eac, ASSIGN in 333 msec 2023-04-08 20:57:41,985 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-08 20:57:41,985 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680987461985"}]},"ts":"1680987461985"} 2023-04-08 20:57:41,986 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-08 20:57:41,987 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-08 20:57:41,988 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1760 sec 2023-04-08 20:57:42,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-08 20:57:42,923 INFO [Listener at localhost.localdomain/41823] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-08 20:57:42,924 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x5054e06c to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:57:42,931 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@67b5579f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:57:42,932 DEBUG [Listener at localhost.localdomain/41823] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:57:42,934 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39318, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:57:42,935 DEBUG [Listener at localhost.localdomain/41823] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-08 20:57:42,938 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38988, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-08 20:57:42,940 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-08 20:57:42,940 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-08 20:57:42,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-08 20:57:43,355 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-08 20:57:43,358 DEBUG [PEWorker-5] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-08 20:57:43,358 INFO [PEWorker-5] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-08 20:57:43,360 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-08 20:57:43,362 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e986ad248e3bbfc4197e05257d0f5eac, REOPEN/MOVE}] 2023-04-08 20:57:43,363 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e986ad248e3bbfc4197e05257d0f5eac, REOPEN/MOVE 2023-04-08 20:57:43,363 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=e986ad248e3bbfc4197e05257d0f5eac, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:43,363 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987463363"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680987463363"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680987463363"}]},"ts":"1680987463363"} 2023-04-08 20:57:43,364 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601}] 2023-04-08 20:57:43,516 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:57:43,517 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing e986ad248e3bbfc4197e05257d0f5eac, disabling compactions & flushes 2023-04-08 20:57:43,517 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:43,517 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:43,517 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. after waiting 0 ms 2023-04-08 20:57:43,517 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:43,521 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-08 20:57:43,522 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:43,522 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for e986ad248e3bbfc4197e05257d0f5eac: 2023-04-08 20:57:43,523 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: e986ad248e3bbfc4197e05257d0f5eac to self. 2023-04-08 20:57:43,525 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:57:43,525 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=e986ad248e3bbfc4197e05257d0f5eac, regionState=CLOSED 2023-04-08 20:57:43,526 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987463525"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680987463525"}]},"ts":"1680987463525"} 2023-04-08 20:57:43,529 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-08 20:57:43,530 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 in 163 msec 2023-04-08 20:57:43,530 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=e986ad248e3bbfc4197e05257d0f5eac, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,41091,1680987313601; forceNewPlan=false, retain=true 2023-04-08 20:57:43,681 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=e986ad248e3bbfc4197e05257d0f5eac, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:43,682 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987463681"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680987463681"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680987463681"}]},"ts":"1680987463681"} 2023-04-08 20:57:43,683 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601}] 2023-04-08 20:57:43,840 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:43,840 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e986ad248e3bbfc4197e05257d0f5eac, NAME => 'TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.', STARTKEY => '', ENDKEY => ''} 2023-04-08 20:57:43,841 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:57:43,841 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 20:57:43,842 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:57:43,842 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:57:43,845 INFO [StoreOpener-e986ad248e3bbfc4197e05257d0f5eac-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:57:43,846 DEBUG [StoreOpener-e986ad248e3bbfc4197e05257d0f5eac-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A 2023-04-08 20:57:43,846 DEBUG [StoreOpener-e986ad248e3bbfc4197e05257d0f5eac-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A 2023-04-08 20:57:43,846 INFO [StoreOpener-e986ad248e3bbfc4197e05257d0f5eac-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 20:57:43,847 INFO [StoreOpener-e986ad248e3bbfc4197e05257d0f5eac-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e986ad248e3bbfc4197e05257d0f5eac columnFamilyName A 2023-04-08 20:57:43,848 INFO [StoreOpener-e986ad248e3bbfc4197e05257d0f5eac-1] regionserver.HStore(310): Store=e986ad248e3bbfc4197e05257d0f5eac/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 20:57:43,849 INFO [StoreOpener-e986ad248e3bbfc4197e05257d0f5eac-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:57:43,850 DEBUG [StoreOpener-e986ad248e3bbfc4197e05257d0f5eac-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B 2023-04-08 20:57:43,850 DEBUG [StoreOpener-e986ad248e3bbfc4197e05257d0f5eac-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B 2023-04-08 20:57:43,850 INFO [StoreOpener-e986ad248e3bbfc4197e05257d0f5eac-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 20:57:43,851 INFO [StoreOpener-e986ad248e3bbfc4197e05257d0f5eac-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e986ad248e3bbfc4197e05257d0f5eac columnFamilyName B 2023-04-08 20:57:43,851 INFO [StoreOpener-e986ad248e3bbfc4197e05257d0f5eac-1] regionserver.HStore(310): Store=e986ad248e3bbfc4197e05257d0f5eac/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 20:57:43,852 INFO [StoreOpener-e986ad248e3bbfc4197e05257d0f5eac-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:57:43,853 DEBUG [StoreOpener-e986ad248e3bbfc4197e05257d0f5eac-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C 2023-04-08 20:57:43,853 DEBUG [StoreOpener-e986ad248e3bbfc4197e05257d0f5eac-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C 2023-04-08 20:57:43,853 INFO [StoreOpener-e986ad248e3bbfc4197e05257d0f5eac-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 20:57:43,853 INFO [StoreOpener-e986ad248e3bbfc4197e05257d0f5eac-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e986ad248e3bbfc4197e05257d0f5eac columnFamilyName C 2023-04-08 20:57:43,854 INFO [StoreOpener-e986ad248e3bbfc4197e05257d0f5eac-1] regionserver.HStore(310): Store=e986ad248e3bbfc4197e05257d0f5eac/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 20:57:43,854 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:43,855 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:57:43,856 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:57:43,859 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-08 20:57:43,861 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:57:43,862 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened e986ad248e3bbfc4197e05257d0f5eac; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69533085, jitterRate=0.03612370789051056}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-08 20:57:43,862 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for e986ad248e3bbfc4197e05257d0f5eac: 2023-04-08 20:57:43,863 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac., pid=61, masterSystemTime=1680987463834 2023-04-08 20:57:43,865 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:43,865 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:43,866 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=e986ad248e3bbfc4197e05257d0f5eac, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:43,866 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987463866"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680987463866"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680987463866"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680987463866"}]},"ts":"1680987463866"} 2023-04-08 20:57:43,870 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-08 20:57:43,870 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 in 185 msec 2023-04-08 20:57:43,873 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-08 20:57:43,873 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e986ad248e3bbfc4197e05257d0f5eac, REOPEN/MOVE in 508 msec 2023-04-08 20:57:43,876 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-08 20:57:43,876 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 514 msec 2023-04-08 20:57:43,878 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 937 msec 2023-04-08 20:57:43,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-08 20:57:43,880 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x323c0ce8 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:57:43,884 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@451cdd43, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:57:43,885 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x0754289a to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:57:43,888 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@55dd2fd4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:57:43,889 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x3d853d1b to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:57:43,892 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1bb3d2c9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:57:43,893 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x112588a6 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:57:43,897 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@55c027c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:57:43,897 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x407f33a2 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:57:43,902 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7d22e3b3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:57:43,903 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x44a601bf to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:57:43,908 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ef34de3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:57:43,908 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x67bf0d7e to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:57:43,913 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@136043cf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:57:43,913 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x3d52316c to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:57:43,922 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@22990f45, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:57:43,922 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x509de0a9 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:57:43,931 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7c686287, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:57:43,932 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(139): Connect 0x17e16418 to 127.0.0.1:52440 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 20:57:43,939 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28c646e5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 20:57:43,949 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:43,950 DEBUG [hconnection-0x2de56d3e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:57:43,955 DEBUG [hconnection-0x647009a7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:57:43,955 DEBUG [hconnection-0x735fc3a6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:57:43,955 DEBUG [hconnection-0x52f798f3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:57:43,956 DEBUG [hconnection-0x7f45345d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:57:43,956 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39348, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:57:43,956 DEBUG [hconnection-0x370362a5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:57:43,957 DEBUG [hconnection-0x5b1a5f84-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:57:43,958 DEBUG [hconnection-0x45ac3fa8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:57:43,958 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39374, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:57:43,959 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39324, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:57:43,959 DEBUG [hconnection-0x40ce9b15-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:57:43,960 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39336, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:57:43,960 DEBUG [hconnection-0x20a20dc5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 20:57:43,961 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39364, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:57:43,961 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39370, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:57:43,963 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39404, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:57:43,964 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39384, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:57:43,967 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39392, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:57:43,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:43,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:43,970 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:43,970 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:43,972 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:43,972 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:43,974 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39408, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 20:57:43,977 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:43,977 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:43,978 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:43,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:43,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:43,978 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:43,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:43,978 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:43,979 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:43,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:43,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:43,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:43,980 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:43,982 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:43,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:57:43,983 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e986ad248e3bbfc4197e05257d0f5eac 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-08 20:57:43,983 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:43,983 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:43,983 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:43,983 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:43,984 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=A 2023-04-08 20:57:43,984 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:43,984 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=B 2023-04-08 20:57:43,984 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:43,985 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=C 2023-04-08 20:57:43,985 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:43,985 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:43,985 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:43,985 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:43,985 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:57:43,985 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:57:43,985 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:43,985 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:43,985 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:43,985 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:43,985 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:43,985 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:43,988 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:43,988 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:43,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:43,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:43,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:43,988 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:43,988 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:43,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:43,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:43,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:43,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:43,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:43,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:43,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13270277[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:43,989 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:43,989 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:43,990 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:43,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:43,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:43,990 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:43,990 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:43,990 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:43,990 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:43,990 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:43,991 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:43,991 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:43,991 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:43,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:43,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:43,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:43,991 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:43,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:43,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:43,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:43,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:43,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:43,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:43,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:43,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:43,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:43,993 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:43,993 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:43,993 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:43,993 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:43,993 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:43,994 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:43,994 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:43,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:43,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:43,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:43,994 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:43,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:43,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:43,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:43,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:43,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:43,994 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:43,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:43,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:43,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:43,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:43,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:43,995 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:43,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:43,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:43,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:43,999 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:43,999 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:43,999 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:43,999 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:43,999 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:43,999 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:43,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:43,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:43,999 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:43,999 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:43,999 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:43,999 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:44,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:44,000 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,000 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:44,000 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,000 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:44,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39324 deadline: 1680987524018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1680987464016, max timestamp=1680987464020Num uniques 859; 2023-04-08 20:57:44,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-08 20:57:44,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680987464016, max timestamp=1680987464020Num uniques 859; 2023-04-08 20:57:44,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-08 20:57:44,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1680987464016, max timestamp=1680987464020Num uniques 859; 2023-04-08 20:57:44,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-08 20:57:44,054 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:44,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39348 deadline: 1680987524051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:44,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39384 deadline: 1680987524052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:44,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39370 deadline: 1680987524052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,056 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:44,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39404 deadline: 1680987524053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,100 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:44,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:44,102 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:44,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:44,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:44,103 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:44,103 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:44,103 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:44,103 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:44,103 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,103 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,104 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,104 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:44,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:44,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:44,105 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,105 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:44,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:44,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,105 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:44,105 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:44,105 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:44,105 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:44,105 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:44,105 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:44,106 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:44,106 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:44,106 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:44,106 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:57:44,106 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:57:44,106 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:44,106 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:44,106 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:44,106 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,106 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:44,106 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:44,107 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,107 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:44,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:44,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:44,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:44,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:44,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:44,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46d70a42[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:44,108 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:44,108 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,109 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,109 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:44,114 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,114 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,114 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:44,114 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:44,114 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:44,114 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:44,115 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,115 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:44,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:44,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:44,115 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:44,115 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:44,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:44,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:44,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:44,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:44,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:44,117 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:44,117 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:44,117 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:44,117 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:44,117 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:44,117 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:44,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:44,117 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,118 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:44,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:44,118 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:44,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:44,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:44,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,119 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:44,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,120 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,120 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,120 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,120 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,120 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:44,120 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:44,120 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:44,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:44,120 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,120 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:44,120 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:44,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:44,120 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,120 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,120 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:44,120 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:44,120 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:44,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:44,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:44,124 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:44,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39324 deadline: 1680987524122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,156 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:44,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39348 deadline: 1680987524155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,157 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:44,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39384 deadline: 1680987524156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:44,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39370 deadline: 1680987524156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,157 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:44,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39404 deadline: 1680987524156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,221 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:44,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:44,222 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:44,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:44,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:44,223 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:44,223 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:44,223 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:44,223 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:44,224 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,224 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,224 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,224 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:44,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:44,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:44,225 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,225 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:44,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:44,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,225 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:44,225 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:44,225 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:44,226 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:44,226 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:44,226 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:44,226 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:44,226 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:44,226 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:44,226 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:57:44,226 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:57:44,226 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:44,226 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:44,226 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:44,226 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,226 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:44,226 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:44,227 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,227 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:44,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:44,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:44,227 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,227 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:44,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:44,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:44,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:44,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3523e28b[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:44,228 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:44,228 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,229 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,229 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:44,229 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:44,229 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,229 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:44,229 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:44,230 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,230 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:44,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:44,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:44,230 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:44,230 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:44,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:44,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:44,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:44,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:44,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:44,231 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:44,231 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:44,231 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:44,232 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:44,232 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:44,232 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,232 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,232 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:44,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:44,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:44,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:44,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:44,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:44,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,234 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:44,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,235 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:44,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,236 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,236 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:44,236 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,236 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:44,236 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:44,236 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:44,236 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:44,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:44,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:44,236 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,236 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,236 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,236 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,236 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:44,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:44,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:44,236 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:44,236 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:44,327 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:44,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39324 deadline: 1680987524326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,336 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:44,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:44,338 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:44,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:44,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:44,339 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:44,339 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:44,339 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:44,339 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:44,340 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,340 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,340 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,340 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:44,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:44,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:44,341 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,341 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:44,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:44,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,341 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:44,341 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:44,342 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:44,342 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:44,342 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:44,342 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:44,342 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:44,342 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:44,342 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:44,342 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:57:44,342 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:57:44,342 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:44,342 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:44,342 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:44,342 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,342 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:44,342 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:44,343 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,343 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:44,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:44,343 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,343 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:44,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:44,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:44,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:44,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:44,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ca6333a[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:44,344 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:44,344 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,345 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,346 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:44,346 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,346 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:44,346 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:44,346 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:44,346 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,346 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:44,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:44,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:44,346 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:44,347 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:44,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:44,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:44,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:44,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:44,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:44,348 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:44,348 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:44,349 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:44,349 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:44,349 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:44,349 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,349 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:44,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:44,349 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:44,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:44,350 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:44,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:44,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:44,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,351 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:44,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,353 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:44,353 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:44,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:44,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:44,353 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,353 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:44,353 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,353 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:44,353 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:44,353 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,353 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,353 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:44,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:44,353 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,353 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,353 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:44,353 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:44,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:44,358 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:44,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39348 deadline: 1680987524357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:44,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39384 deadline: 1680987524357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,359 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:44,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39370 deadline: 1680987524358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,359 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:44,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39404 deadline: 1680987524358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,397 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040887f0f8c94b3140c581f1d93ec193719b to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040887f0f8c94b3140c581f1d93ec193719b 2023-04-08 20:57:44,453 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:44,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:44,455 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:44,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:44,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:44,456 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:44,456 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:44,456 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:44,456 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:44,457 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,457 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,457 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:44,457 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:44,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:44,457 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,458 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:44,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:44,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,458 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:44,459 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:44,459 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:44,459 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:44,459 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:44,459 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:44,459 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:44,459 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:44,459 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:44,459 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:57:44,459 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:57:44,459 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:44,459 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:44,459 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:44,459 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,459 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:44,459 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:44,460 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,460 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:44,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:44,460 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,460 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:44,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:44,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:44,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:44,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:44,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6739b463[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:44,461 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:44,461 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,462 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,462 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,462 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:44,462 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:44,462 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:44,462 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:44,462 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:44,462 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,462 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:44,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:44,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:44,463 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:44,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:44,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:44,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:44,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:44,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:44,464 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:44,464 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:44,464 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:44,464 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:44,464 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:44,465 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:44,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:44,465 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,465 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:44,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:44,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:44,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:44,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,466 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:44,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:44,467 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,469 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,469 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:44,469 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,469 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:44,469 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,469 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:44,469 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:44,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:44,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:44,469 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,469 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:44,469 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:44,469 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:44,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:44,469 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:44,469 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,469 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,569 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:44,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:44,571 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:44,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:44,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:44,572 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:44,572 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:44,572 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:44,572 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:44,572 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,572 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,573 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,573 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:44,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:44,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:44,573 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,573 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:44,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:44,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,574 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:44,574 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:44,574 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:44,574 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:44,574 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:44,574 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:44,574 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:44,574 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:44,574 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:57:44,574 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:57:44,574 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:44,574 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:44,574 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:44,574 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,574 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:44,574 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:44,575 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,575 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:44,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:44,575 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,575 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:44,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:44,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:44,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:44,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:44,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19c237dc[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:44,576 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:44,576 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,577 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,577 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:44,577 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:44,577 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,577 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:44,577 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:44,578 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,578 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:44,578 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:44,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:44,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:44,578 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:44,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:44,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:44,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:44,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:44,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:44,579 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:44,579 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:44,579 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:44,579 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:44,579 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:44,580 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,580 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,580 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:44,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:44,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:44,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,580 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:44,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:44,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:44,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:44,581 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:44,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,583 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,583 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:44,583 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,583 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:44,583 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,583 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:44,583 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:44,583 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:44,583 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:44,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:44,583 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:44,583 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,583 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:44,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:44,583 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,583 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:44,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:44,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:44,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39324 deadline: 1680987524628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:44,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39348 deadline: 1680987524659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:44,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39384 deadline: 1680987524661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:44,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39370 deadline: 1680987524661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:44,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39404 deadline: 1680987524662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,683 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:44,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:44,685 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:44,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:44,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:44,686 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:44,686 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:44,686 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:44,686 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:44,686 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,686 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,687 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:44,687 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:44,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:44,687 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,687 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:44,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:44,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,688 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:44,688 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:44,688 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:44,688 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:44,688 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:44,688 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:44,688 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:44,688 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:44,688 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:44,688 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:57:44,689 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:57:44,689 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:44,689 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:44,689 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:44,689 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,689 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:44,689 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:44,690 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,690 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:44,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:44,690 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,690 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:44,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:44,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:44,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:44,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:44,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33dfa9d5[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:44,691 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:44,691 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,691 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,692 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:44,692 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:44,692 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,692 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:44,692 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:44,692 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,692 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:44,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:44,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:44,692 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:44,693 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:44,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:44,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:44,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:44,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:44,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:44,694 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:44,694 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:44,694 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:44,694 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:44,694 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:44,695 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,695 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:44,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,695 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:44,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:44,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:44,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:44,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:44,696 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:44,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,697 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:44,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,698 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:44,698 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:44,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:44,698 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:44,698 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:44,698 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:44,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:44,698 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:44,698 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,698 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,698 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,698 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,699 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,699 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:44,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:44,698 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:44,699 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:44,799 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:44,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:44,800 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:44,801 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/A/8c7834a3e2b0497fb62833f485237746 2023-04-08 20:57:44,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:44,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:44,801 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:44,801 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:44,801 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:44,801 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:44,802 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,802 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,802 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,802 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:44,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:44,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:44,803 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,803 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:44,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:44,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,804 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:44,804 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:44,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:44,804 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:44,804 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:44,804 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:44,805 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:44,805 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:44,805 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:44,805 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:57:44,805 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:57:44,805 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:44,805 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:44,805 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:44,805 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,805 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:44,805 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:44,806 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:44,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:44,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:44,807 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:44,807 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,807 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:44,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:44,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:44,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e9d6377[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:44,808 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:44,808 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,815 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,815 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:44,815 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,815 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:44,815 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:44,815 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:44,816 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,816 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:44,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:44,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:44,816 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:44,816 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:44,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:44,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:44,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:44,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:44,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:44,818 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:44,818 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:44,818 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:44,818 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:44,818 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:44,819 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,819 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,819 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:44,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:44,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:44,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:44,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:44,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:44,819 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:44,820 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:44,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,822 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,822 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,822 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,822 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,822 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:44,822 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:44,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:44,822 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:44,822 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:44,822 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:44,822 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,822 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,822 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:44,822 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:44,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:44,822 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:44,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:44,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:44,922 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:44,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:44,924 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:44,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:44,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:44,925 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:44,925 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:44,925 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:44,925 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:44,926 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,926 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,926 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,926 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:44,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:44,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:44,927 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,927 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:44,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:44,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,927 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:44,927 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:44,927 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:44,927 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:44,927 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:44,927 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:44,927 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:44,928 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:44,928 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:44,928 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:57:44,928 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:57:44,928 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:44,928 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:44,928 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:44,928 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,928 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:44,928 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:44,928 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,928 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:44,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:44,929 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,929 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:44,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:44,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:44,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:44,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:44,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@642d4c6b[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:44,930 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:44,930 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,930 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,930 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:44,930 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:44,930 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:44,931 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,931 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:44,931 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,931 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:44,931 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:44,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:44,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:44,931 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:44,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:44,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:44,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:44,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:44,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:44,933 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:44,933 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:44,933 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:44,933 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:44,933 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:44,934 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,934 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:44,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:44,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,934 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:44,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:44,934 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:44,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:44,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:44,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,935 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:44,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:44,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,936 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,937 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:44,937 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,937 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:44,937 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:44,937 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:44,937 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:44,937 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:44,937 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,937 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:44,937 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:44,937 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:44,937 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:44,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:44,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:44,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:44,937 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:44,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:44,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:45,037 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:45,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:45,039 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:45,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:45,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:45,040 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:45,040 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:45,040 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:45,040 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:45,040 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,040 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:45,041 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:45,041 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,041 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,041 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:45,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:45,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:45,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:45,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:45,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:45,043 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:45,043 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:45,043 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:45,043 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:45,043 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:45,043 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:45,043 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:45,043 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:45,043 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:45,043 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:57:45,043 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:57:45,043 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:45,044 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:45,044 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:45,044 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,044 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:45,044 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:45,044 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,044 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:45,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:45,044 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,044 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:45,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:45,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:45,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:45,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:45,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:45,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cd9fbe5[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:45,046 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:45,046 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,046 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,046 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:45,046 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:45,046 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:45,046 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,046 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:45,047 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:45,047 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:45,047 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:45,047 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:45,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:45,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:45,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:45,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:45,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:45,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:45,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:45,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:45,049 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:45,049 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:45,049 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:45,049 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:45,049 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:45,050 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,050 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,050 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:45,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:45,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:45,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:45,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:45,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:45,051 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:45,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:45,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:45,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,052 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:45,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:45,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,053 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,053 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:45,053 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:45,053 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:45,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:45,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:45,053 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:45,053 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:45,053 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:45,053 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:45,053 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:45,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:45,053 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,054 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,054 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:45,054 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,054 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:45,135 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:45,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39324 deadline: 1680987525134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,154 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:45,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:45,155 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:45,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:45,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:45,156 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:45,156 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:45,156 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:45,156 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:45,157 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,157 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:45,157 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:45,157 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:45,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:45,158 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,158 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:45,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:45,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:45,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:45,158 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:45,158 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:45,158 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:45,159 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:45,159 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:45,159 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:45,159 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:45,159 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:45,159 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:45,159 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:57:45,159 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:57:45,159 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:45,160 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:45,160 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:45,160 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,160 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:45,160 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:45,161 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,161 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:45,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:45,161 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,161 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:45,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:45,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:45,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:45,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:45,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:45,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@220666aa[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:45,162 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:45,162 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,163 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,163 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:45,163 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:45,163 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:45,163 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,163 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:45,164 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:45,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39348 deadline: 1680987525162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,165 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,165 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:45,165 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:45,165 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:45,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:45,167 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:45,167 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:45,167 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:45,167 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:45,167 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:45,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:45,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39370 deadline: 1680987525165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:45,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39404 deadline: 1680987525165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,168 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,168 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:45,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:45,168 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:45,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:45,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:45,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:45,168 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:45,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:45,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:45,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,169 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,169 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:45,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39384 deadline: 1680987525167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:45,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:45,170 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,170 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:45,170 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:45,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:45,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:45,171 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:45,170 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:45,171 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:45,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:45,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:45,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:45,171 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:45,171 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:45,171 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,171 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,171 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,171 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:45,171 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,171 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:45,210 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/B/72dc29331ded40d08416a8a884400312 2023-04-08 20:57:45,271 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:45,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:45,273 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:45,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:45,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:45,274 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:45,274 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:45,274 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:45,274 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:45,274 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,275 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:45,275 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,275 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:45,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:45,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:45,275 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,275 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:45,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:45,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:45,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:45,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:45,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:45,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:45,276 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:45,276 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:45,276 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:45,277 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:45,277 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:45,277 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:45,277 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:57:45,277 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:57:45,277 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:45,277 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:45,277 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:45,277 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,277 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:45,277 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:45,278 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:45,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:45,278 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:45,279 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,279 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:45,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:45,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:45,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:45,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:45,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a53f46[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:45,280 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:45,280 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,281 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,281 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:45,281 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,281 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:45,281 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:45,281 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:45,281 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,281 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:45,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:45,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:45,282 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:45,282 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:45,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:45,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:45,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:45,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:45,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:45,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:45,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:45,283 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:45,283 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:45,283 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:45,283 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:45,284 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:45,284 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,284 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,284 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:45,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:45,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:45,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,285 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:45,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:45,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:45,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:45,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:45,286 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:45,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:45,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:45,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,287 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:45,287 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,287 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:45,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:45,287 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:45,287 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:45,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:45,287 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:45,287 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:45,288 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:45,288 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:45,288 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:45,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:45,288 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,290 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,290 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,290 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:45,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:45,388 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:45,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:45,389 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:45,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:45,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:45,391 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:45,391 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:45,391 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:45,391 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:45,391 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,391 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:45,392 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:45,392 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:45,392 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,392 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:45,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:45,392 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:45,393 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:45,393 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:45,393 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:45,393 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:45,393 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:45,393 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:45,393 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:45,393 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:45,393 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:57:45,393 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:57:45,393 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:45,393 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:45,393 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:45,393 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,393 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:45,393 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:45,394 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,394 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:45,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:45,395 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,395 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:45,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:45,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:45,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:45,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:45,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:45,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@96565cd[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:45,396 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:45,396 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,396 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,396 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:45,396 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:45,396 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:45,396 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,396 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:45,397 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,397 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:45,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:45,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:45,397 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:45,397 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:45,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:45,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:45,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:45,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:45,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:45,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:45,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:45,399 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:45,399 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:45,399 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:45,399 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:45,399 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:45,400 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,400 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:45,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:45,400 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:45,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:45,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:45,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:45,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,401 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:45,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:45,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:45,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,402 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:45,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:45,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,403 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,403 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:45,403 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:45,403 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:45,403 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:45,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:45,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:45,403 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:45,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:45,403 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,403 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:45,403 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:45,403 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,403 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,403 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,403 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:45,404 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:45,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:45,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:45,504 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:45,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:45,505 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:45,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:45,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:45,506 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:45,506 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:45,506 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:45,506 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:45,507 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,507 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:45,507 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,507 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:45,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:45,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:45,508 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:45,508 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:45,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:45,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:45,508 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:45,509 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:45,509 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:45,509 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:45,509 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:45,509 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:45,509 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:45,509 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:45,509 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:45,509 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:57:45,509 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:57:45,509 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:45,509 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:45,509 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:45,509 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,509 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:45,509 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:45,510 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,510 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:45,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:45,510 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,510 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:45,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:45,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:45,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:45,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:45,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:45,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ac0a1d7[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:45,511 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:45,511 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,512 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,512 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:45,512 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:45,512 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:45,512 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,512 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:45,513 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:45,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:45,513 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:45,513 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:45,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:45,513 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:45,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:45,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:45,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:45,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:45,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:45,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:45,514 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:45,514 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:45,515 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:45,515 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:45,515 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:45,516 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,516 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:45,516 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:45,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:45,516 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:45,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:45,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:45,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:45,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:45,517 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:45,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,518 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:45,518 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,518 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:45,518 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:45,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:45,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:45,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:45,518 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:45,519 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:45,518 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:45,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:45,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:45,519 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:45,519 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,519 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,519 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,519 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,519 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:45,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:45,519 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:45,619 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/C/e65543d837f940a49870fed5e1ed360b 2023-04-08 20:57:45,619 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:45,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:45,621 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:45,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:45,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:45,622 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:45,622 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:45,622 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:45,622 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:45,622 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,622 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:45,623 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/A/8c7834a3e2b0497fb62833f485237746 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/8c7834a3e2b0497fb62833f485237746 2023-04-08 20:57:45,623 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,623 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:45,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:45,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:45,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:45,623 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,624 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:45,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:45,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:45,624 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:45,624 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:45,624 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:45,624 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:45,624 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:45,624 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:45,624 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:45,624 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:45,624 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:45,624 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:57:45,625 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:57:45,625 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:45,625 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:45,625 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:45,625 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,625 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:45,625 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:45,625 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/8c7834a3e2b0497fb62833f485237746, entries=150, sequenceid=16, filesize=25.1 K 2023-04-08 20:57:45,626 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,626 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,626 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,626 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:45,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:45,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:45,626 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/B/72dc29331ded40d08416a8a884400312 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/72dc29331ded40d08416a8a884400312 2023-04-08 20:57:45,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:45,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:45,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:45,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:45,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:45,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3be2a113[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:45,628 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:45,628 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,628 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,628 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:45,628 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:45,628 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:45,629 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,629 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:45,629 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/72dc29331ded40d08416a8a884400312, entries=150, sequenceid=16, filesize=11.6 K 2023-04-08 20:57:45,629 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,629 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:45,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:45,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:45,629 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:45,629 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:45,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:45,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:45,630 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/C/e65543d837f940a49870fed5e1ed360b as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/e65543d837f940a49870fed5e1ed360b 2023-04-08 20:57:45,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:45,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:45,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:45,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:45,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:45,631 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:45,631 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:45,631 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:45,631 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:45,631 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:45,632 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,632 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:45,632 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:45,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:45,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,632 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/e65543d837f940a49870fed5e1ed360b, entries=150, sequenceid=16, filesize=11.6 K 2023-04-08 20:57:45,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:45,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:45,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:45,633 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,633 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=127.47 KB/130530 for e986ad248e3bbfc4197e05257d0f5eac in 1650ms, sequenceid=16, compaction requested=false 2023-04-08 20:57:45,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:45,633 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e986ad248e3bbfc4197e05257d0f5eac: 2023-04-08 20:57:45,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:45,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:45,634 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:45,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:45,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,636 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-08 20:57:45,636 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:45,636 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:45,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:45,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:45,636 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:45,637 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:45,637 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:45,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:45,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:45,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:45,637 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:45,637 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:45,646 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,646 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:45,646 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:45,646 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,646 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,646 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:45,646 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,737 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:45,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:45,739 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:45,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:45,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:45,740 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:45,740 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:45,740 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:45,740 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:45,740 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,740 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:45,741 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,741 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:45,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:45,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:45,741 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:45,741 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:45,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:45,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:45,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:45,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:45,742 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:45,742 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:45,742 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:45,742 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:45,742 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:45,742 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:45,746 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:45,746 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:45,746 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:45,746 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:57:45,746 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e986ad248e3bbfc4197e05257d0f5eac 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-08 20:57:45,746 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=A 2023-04-08 20:57:45,746 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:57:45,746 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=B 2023-04-08 20:57:45,746 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:57:45,746 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=C 2023-04-08 20:57:45,746 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:57:46,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:57:46,145 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:57:46,162 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:46,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39324 deadline: 1680987526161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:46,169 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040896c416f74dc84b9185f764634becf76d to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040896c416f74dc84b9185f764634becf76d 2023-04-08 20:57:46,171 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:46,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39370 deadline: 1680987526170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:46,173 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:46,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39384 deadline: 1680987526171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:46,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:46,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39348 deadline: 1680987526173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:46,177 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:46,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39404 deadline: 1680987526176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:46,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:46,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39324 deadline: 1680987526263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:46,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:46,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39324 deadline: 1680987526466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:46,573 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/A/39fbfc5b9f1d45db951c704a49236afe 2023-04-08 20:57:46,770 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:46,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39324 deadline: 1680987526769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:46,983 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/B/cd573042af1842a7a9c4a5fb3e5bc6fc 2023-04-08 20:57:47,275 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:47,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39324 deadline: 1680987527274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:47,392 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/C/1837a5958ab34dcd8b72ab59fcc44e27 2023-04-08 20:57:47,395 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/A/39fbfc5b9f1d45db951c704a49236afe as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/39fbfc5b9f1d45db951c704a49236afe 2023-04-08 20:57:47,398 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/39fbfc5b9f1d45db951c704a49236afe, entries=150, sequenceid=50, filesize=25.1 K 2023-04-08 20:57:47,399 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/B/cd573042af1842a7a9c4a5fb3e5bc6fc as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/cd573042af1842a7a9c4a5fb3e5bc6fc 2023-04-08 20:57:47,402 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/cd573042af1842a7a9c4a5fb3e5bc6fc, entries=150, sequenceid=50, filesize=11.6 K 2023-04-08 20:57:47,402 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/C/1837a5958ab34dcd8b72ab59fcc44e27 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/1837a5958ab34dcd8b72ab59fcc44e27 2023-04-08 20:57:47,405 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/1837a5958ab34dcd8b72ab59fcc44e27, entries=150, sequenceid=50, filesize=11.6 K 2023-04-08 20:57:47,405 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for e986ad248e3bbfc4197e05257d0f5eac in 1659ms, sequenceid=50, compaction requested=false 2023-04-08 20:57:47,405 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e986ad248e3bbfc4197e05257d0f5eac: 2023-04-08 20:57:47,405 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:47,405 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:47,405 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:47,405 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:47,406 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:47,406 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:47,406 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:47,406 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:47,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:47,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:47,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:47,407 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:47,407 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:47,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:47,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:47,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:47,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:47,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:47,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:47,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1593f626[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:47,408 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:47,408 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:47,409 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:47,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:47,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:47,409 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:47,409 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:47,409 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:47,409 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:47,409 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:47,410 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:47,410 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:47,410 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:47,410 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:47,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:47,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:47,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:47,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:47,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:47,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:47,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:47,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:47,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:47,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:47,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:47,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:47,412 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:47,412 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:47,412 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:47,412 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:47,412 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:47,412 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:47,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:47,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:47,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:47,412 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:47,413 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:47,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:47,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:47,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:47,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:47,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:47,413 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:47,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:47,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:47,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:47,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:47,414 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:47,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:47,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:47,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:47,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:47,415 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:47,415 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:47,415 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:47,415 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:47,415 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:47,415 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:47,415 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:47,415 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:47,415 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:47,416 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:47,415 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:47,416 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:47,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:47,416 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:47,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:47,416 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:47,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:47,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:47,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:47,516 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:47,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:47,518 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:47,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:47,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:47,518 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:47,518 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:47,519 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:47,519 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:47,519 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:47,519 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:47,519 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:47,519 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:47,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:47,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:47,520 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:47,520 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:47,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:47,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:47,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:47,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:47,524 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:47,524 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:47,524 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:47,524 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:47,524 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:47,524 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:47,525 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:47,525 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:47,525 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:47,525 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:57:47,525 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e986ad248e3bbfc4197e05257d0f5eac 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-08 20:57:47,525 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=A 2023-04-08 20:57:47,525 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:47,525 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=B 2023-04-08 20:57:47,525 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:47,525 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=C 2023-04-08 20:57:47,525 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:47,936 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408c1465650c5c7444aa2fc78f06cb71aed to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408c1465650c5c7444aa2fc78f06cb71aed 2023-04-08 20:57:48,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:57:48,176 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:57:48,197 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:48,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39348 deadline: 1680987528194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:48,197 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:48,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39370 deadline: 1680987528195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:48,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680987468186, max timestamp=1680987468196Num uniques 859; 2023-04-08 20:57:48,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680987468186, max timestamp=1680987468196Num uniques 859; 2023-04-08 20:57:48,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-08 20:57:48,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-08 20:57:48,198 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680987468186, max timestamp=1680987468196Num uniques 859; 2023-04-08 20:57:48,198 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-08 20:57:48,224 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:48,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39404 deadline: 1680987528222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:48,226 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:48,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39384 deadline: 1680987528224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:48,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:48,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39324 deadline: 1680987528276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:48,299 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:48,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39348 deadline: 1680987528298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:48,299 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:48,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39370 deadline: 1680987528298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:48,326 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:48,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39404 deadline: 1680987528325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:48,329 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:48,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39384 deadline: 1680987528327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:48,340 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=61, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/A/b7a7fae04c134dcab05eeacc02a6266a 2023-04-08 20:57:48,501 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:48,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39348 deadline: 1680987528500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:48,501 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:48,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39370 deadline: 1680987528500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:48,529 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:48,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39404 deadline: 1680987528527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:48,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:48,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39384 deadline: 1680987528530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:48,747 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/B/0de7c8d866c041209911a04fe1c18192 2023-04-08 20:57:48,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:48,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39348 deadline: 1680987528802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:48,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:48,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39370 deadline: 1680987528803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:48,831 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:48,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39404 deadline: 1680987528830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:48,837 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:48,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39384 deadline: 1680987528835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:49,155 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/C/d7cca22e678a4c828c1d204a48518955 2023-04-08 20:57:49,158 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/A/b7a7fae04c134dcab05eeacc02a6266a as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/b7a7fae04c134dcab05eeacc02a6266a 2023-04-08 20:57:49,161 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/b7a7fae04c134dcab05eeacc02a6266a, entries=150, sequenceid=61, filesize=25.1 K 2023-04-08 20:57:49,161 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/B/0de7c8d866c041209911a04fe1c18192 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/0de7c8d866c041209911a04fe1c18192 2023-04-08 20:57:49,164 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/0de7c8d866c041209911a04fe1c18192, entries=150, sequenceid=61, filesize=11.6 K 2023-04-08 20:57:49,164 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/C/d7cca22e678a4c828c1d204a48518955 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/d7cca22e678a4c828c1d204a48518955 2023-04-08 20:57:49,167 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/d7cca22e678a4c828c1d204a48518955, entries=150, sequenceid=61, filesize=11.6 K 2023-04-08 20:57:49,167 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for e986ad248e3bbfc4197e05257d0f5eac in 1642ms, sequenceid=61, compaction requested=true 2023-04-08 20:57:49,167 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e986ad248e3bbfc4197e05257d0f5eac: 2023-04-08 20:57:49,167 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:49,168 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:49,168 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:49,168 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:49,168 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:49,168 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:49,169 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:49,169 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:49,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:49,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:49,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:49,169 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:49,169 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:49,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:49,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:49,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:49,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:49,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:49,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:49,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@377ab254[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:49,170 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:49,170 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:49,170 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:49,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:49,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:49,171 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:49,171 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:49,171 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:49,171 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:49,171 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:49,172 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:49,172 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:49,172 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:49,172 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:49,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:49,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:49,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:49,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:49,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:49,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:49,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:49,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:49,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:49,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:49,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:49,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:49,174 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:49,174 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:49,174 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:49,174 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:49,174 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:49,175 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:49,175 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:49,175 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:49,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:49,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:49,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:49,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:49,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:49,175 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:49,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:49,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:49,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:49,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:49,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:49,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:49,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:49,176 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:49,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:49,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:49,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:49,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:49,178 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:49,178 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:49,178 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:49,178 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:49,178 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:49,178 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:49,178 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:49,178 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:49,178 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:49,178 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:49,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:49,178 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:49,178 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:49,178 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:49,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:49,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:49,178 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:49,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:49,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:49,278 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:49,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:49,280 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:49,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:49,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:49,282 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:49,282 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:49,282 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:49,282 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:49,282 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:49,282 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:49,283 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:49,285 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:49,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:49,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:49,285 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:49,285 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:49,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:49,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:49,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:49,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:49,285 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:49,286 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:49,286 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:49,286 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:49,286 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:49,286 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:49,286 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:49,286 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:49,286 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:49,286 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:57:49,286 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e986ad248e3bbfc4197e05257d0f5eac 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-08 20:57:49,286 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=A 2023-04-08 20:57:49,286 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:57:49,287 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=B 2023-04-08 20:57:49,287 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:57:49,287 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=C 2023-04-08 20:57:49,287 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:57:49,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:57:49,306 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:57:49,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:49,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39348 deadline: 1680987529318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:49,324 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:49,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39370 deadline: 1680987529321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:49,333 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:49,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39404 deadline: 1680987529332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:49,343 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:49,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39384 deadline: 1680987529341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:49,424 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:49,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39348 deadline: 1680987529423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:49,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:49,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39370 deadline: 1680987529425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:49,627 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:49,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39348 deadline: 1680987529626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:49,631 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:49,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39370 deadline: 1680987529629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:49,697 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040880b738c7d0a6499d8f63996a1991c447 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040880b738c7d0a6499d8f63996a1991c447 2023-04-08 20:57:49,932 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:49,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39348 deadline: 1680987529929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:49,932 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:49,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39370 deadline: 1680987529932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:50,106 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=96, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/A/1df5438db2c84d41a1bde13f7f90d8b6 2023-04-08 20:57:50,281 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:50,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39324 deadline: 1680987530279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:50,282 DEBUG [Thread-3374] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4121 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac., hostname=jenkins-hbase20.apache.org,41091,1680987313601, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 20:57:50,340 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:50,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39404 deadline: 1680987530338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:50,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:50,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39384 deadline: 1680987530351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:50,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:50,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39370 deadline: 1680987530436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:50,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:50,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39348 deadline: 1680987530437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:50,514 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/B/7fab73ecc11f44e8a2fba15761dede3c 2023-04-08 20:57:50,923 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/C/ecc7240eb5264597bbe3a85160fab2cf 2023-04-08 20:57:50,926 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/A/1df5438db2c84d41a1bde13f7f90d8b6 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/1df5438db2c84d41a1bde13f7f90d8b6 2023-04-08 20:57:50,929 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/1df5438db2c84d41a1bde13f7f90d8b6, entries=150, sequenceid=96, filesize=25.1 K 2023-04-08 20:57:50,929 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/B/7fab73ecc11f44e8a2fba15761dede3c as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/7fab73ecc11f44e8a2fba15761dede3c 2023-04-08 20:57:50,932 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/7fab73ecc11f44e8a2fba15761dede3c, entries=150, sequenceid=96, filesize=11.6 K 2023-04-08 20:57:50,932 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/C/ecc7240eb5264597bbe3a85160fab2cf as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/ecc7240eb5264597bbe3a85160fab2cf 2023-04-08 20:57:50,935 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/ecc7240eb5264597bbe3a85160fab2cf, entries=150, sequenceid=96, filesize=11.6 K 2023-04-08 20:57:50,936 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for e986ad248e3bbfc4197e05257d0f5eac in 1650ms, sequenceid=96, compaction requested=true 2023-04-08 20:57:50,936 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e986ad248e3bbfc4197e05257d0f5eac: 2023-04-08 20:57:50,936 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:50,936 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:50,936 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:50,936 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:50,936 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:50,936 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:50,937 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:50,937 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:50,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:50,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:50,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:50,939 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:50,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:50,939 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:50,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:50,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:50,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:50,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:50,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:50,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@189b7375[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:50,940 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:50,940 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:50,941 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:50,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:50,941 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:50,941 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:50,941 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:50,941 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:50,941 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:50,942 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:50,942 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:50,942 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:50,942 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:50,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:50,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:50,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:50,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:50,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:50,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:50,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:50,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:50,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:50,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:50,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:50,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:50,944 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:50,944 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:50,944 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:50,944 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:50,944 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:50,945 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:50,945 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:50,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:50,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:50,945 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:50,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:50,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:50,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:50,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:50,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:50,945 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:50,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:50,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:50,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:50,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:50,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:50,946 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:50,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:50,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:50,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:50,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:50,947 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:50,947 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:50,947 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:50,947 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:50,947 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:50,947 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:50,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:50,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:50,947 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:50,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:50,947 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:50,948 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:50,947 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:50,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:50,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:50,948 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:50,948 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:50,948 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:50,948 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:51,048 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:51,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:51,050 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:51,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:51,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:51,051 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:51,051 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:51,051 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:51,051 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:51,051 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:51,051 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:51,052 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:51,052 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:51,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:51,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:51,052 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:51,052 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:51,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:51,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:51,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:51,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:51,053 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:51,053 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:51,053 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:51,053 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:51,053 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:51,053 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:51,054 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:51,054 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:51,054 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:51,054 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:57:51,054 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e986ad248e3bbfc4197e05257d0f5eac 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-08 20:57:51,054 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=A 2023-04-08 20:57:51,054 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:51,054 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=B 2023-04-08 20:57:51,054 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:51,054 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=C 2023-04-08 20:57:51,054 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:51,066 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304087c71218294074bada12f5fa41b3e88ae to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304087c71218294074bada12f5fa41b3e88ae 2023-04-08 20:57:51,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:57:51,441 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:57:51,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:51,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39348 deadline: 1680987531467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:51,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680987471462, max timestamp=1680987471468Num uniques 859; 2023-04-08 20:57:51,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 20:57:51,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680987471462, max timestamp=1680987471468Num uniques 859; 2023-04-08 20:57:51,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680987471462, max timestamp=1680987471468Num uniques 859; 2023-04-08 20:57:51,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 20:57:51,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 20:57:51,471 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=107, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/A/b41833a47369456cb3bacb40b070d410 2023-04-08 20:57:51,532 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:51,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39370 deadline: 1680987531530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:51,574 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:51,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39348 deadline: 1680987531572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:51,635 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:51,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39370 deadline: 1680987531633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:51,776 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:51,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39348 deadline: 1680987531775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:51,839 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:51,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39370 deadline: 1680987531837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:51,879 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/B/7c91e95173484b28886611bdee7822fb 2023-04-08 20:57:51,887 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/C/7329a7d4e110480e85f639e9714bcb88 2023-04-08 20:57:51,890 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/A/b41833a47369456cb3bacb40b070d410 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/b41833a47369456cb3bacb40b070d410 2023-04-08 20:57:51,893 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/b41833a47369456cb3bacb40b070d410, entries=150, sequenceid=107, filesize=25.1 K 2023-04-08 20:57:51,894 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/B/7c91e95173484b28886611bdee7822fb as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/7c91e95173484b28886611bdee7822fb 2023-04-08 20:57:51,896 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/7c91e95173484b28886611bdee7822fb, entries=150, sequenceid=107, filesize=11.6 K 2023-04-08 20:57:51,897 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/C/7329a7d4e110480e85f639e9714bcb88 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/7329a7d4e110480e85f639e9714bcb88 2023-04-08 20:57:51,899 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/7329a7d4e110480e85f639e9714bcb88, entries=150, sequenceid=107, filesize=11.6 K 2023-04-08 20:57:51,900 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for e986ad248e3bbfc4197e05257d0f5eac in 846ms, sequenceid=107, compaction requested=true 2023-04-08 20:57:51,900 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e986ad248e3bbfc4197e05257d0f5eac: 2023-04-08 20:57:51,900 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:51,900 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:51,900 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:51,900 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:51,900 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:51,900 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:51,901 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:51,901 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:51,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:51,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:51,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:51,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:51,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:51,902 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:51,902 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:51,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:51,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:51,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:51,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:51,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@365fb34c[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:51,902 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:51,902 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:51,903 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:51,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:51,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:51,903 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:51,903 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:51,903 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:51,903 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:51,903 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:51,904 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:51,904 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:51,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:51,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:51,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:51,904 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:51,904 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:51,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:51,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:51,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:51,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:51,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:51,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:51,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:51,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:51,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:51,906 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:51,906 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:51,906 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:51,906 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:51,906 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:51,907 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:51,907 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:51,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:51,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:51,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:51,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:51,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:51,907 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:51,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:51,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:51,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:51,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:51,908 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:51,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:51,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:51,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:51,908 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:51,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:51,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:51,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:51,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:51,910 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:51,910 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:51,910 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:51,910 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:51,910 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:51,910 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:51,910 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:51,910 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:51,910 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:51,910 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:51,910 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:51,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:51,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:51,910 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:51,910 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:51,910 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:51,910 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:51,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:51,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:52,010 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:52,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:52,012 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:52,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:52,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:52,013 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:52,013 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:52,013 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:52,013 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:52,013 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:52,013 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:52,014 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:52,014 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:52,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:52,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:52,014 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:52,014 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:52,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:52,015 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:52,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:52,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:52,015 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:52,015 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:52,015 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:52,015 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:52,015 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:52,015 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:52,015 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:52,015 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:52,015 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:52,016 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:57:52,016 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e986ad248e3bbfc4197e05257d0f5eac 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-08 20:57:52,016 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=A 2023-04-08 20:57:52,016 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:57:52,016 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=B 2023-04-08 20:57:52,016 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:57:52,016 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=C 2023-04-08 20:57:52,016 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:57:52,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:57:52,082 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:57:52,110 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:52,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39348 deadline: 1680987532106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:52,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:52,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39370 deadline: 1680987532142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:52,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:52,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39348 deadline: 1680987532211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:52,356 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:52,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39404 deadline: 1680987532355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:52,358 DEBUG [Thread-3368] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4135 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac., hostname=jenkins-hbase20.apache.org,41091,1680987313601, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 20:57:52,372 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:52,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39384 deadline: 1680987532371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:52,373 DEBUG [Thread-3366] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4149 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac., hostname=jenkins-hbase20.apache.org,41091,1680987313601, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 20:57:52,414 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:52,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39348 deadline: 1680987532413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:52,439 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408b9b5fc464d674a7a9519e4e1084f7022 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408b9b5fc464d674a7a9519e4e1084f7022 2023-04-08 20:57:52,443 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=143, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/A/89ef2fd6787b4922a37291f8539ca12c 2023-04-08 20:57:52,649 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:52,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39370 deadline: 1680987532647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:52,717 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:52,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39348 deadline: 1680987532715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:52,861 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/B/fdde235952c64d7193f5e601145b538b 2023-04-08 20:57:53,220 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:53,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39348 deadline: 1680987533220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:53,270 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/C/fafdf2a87caf4e2588ef70fcff4855ee 2023-04-08 20:57:53,273 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/A/89ef2fd6787b4922a37291f8539ca12c as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/89ef2fd6787b4922a37291f8539ca12c 2023-04-08 20:57:53,276 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/89ef2fd6787b4922a37291f8539ca12c, entries=150, sequenceid=143, filesize=25.3 K 2023-04-08 20:57:53,276 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/B/fdde235952c64d7193f5e601145b538b as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/fdde235952c64d7193f5e601145b538b 2023-04-08 20:57:53,279 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/fdde235952c64d7193f5e601145b538b, entries=150, sequenceid=143, filesize=11.8 K 2023-04-08 20:57:53,280 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/C/fafdf2a87caf4e2588ef70fcff4855ee as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/fafdf2a87caf4e2588ef70fcff4855ee 2023-04-08 20:57:53,282 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/fafdf2a87caf4e2588ef70fcff4855ee, entries=150, sequenceid=143, filesize=11.8 K 2023-04-08 20:57:53,283 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for e986ad248e3bbfc4197e05257d0f5eac in 1267ms, sequenceid=143, compaction requested=true 2023-04-08 20:57:53,283 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e986ad248e3bbfc4197e05257d0f5eac: 2023-04-08 20:57:53,283 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:53,283 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:53,283 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:53,283 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:53,286 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:53,286 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:53,286 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:53,286 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:53,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:53,287 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:53,287 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:53,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:53,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:53,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cf74167[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:53,288 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:53,288 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:53,288 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:53,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:53,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:53,288 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:53,288 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:53,288 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:53,288 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:53,288 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:53,289 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:53,289 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:53,289 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:53,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:53,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:53,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:53,289 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:53,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:53,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:53,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:53,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:53,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:53,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:53,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:53,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:53,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:53,291 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:53,291 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:53,291 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:53,291 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:53,291 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:53,292 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:53,292 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:53,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:53,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:53,292 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:53,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:53,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:53,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:53,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:53,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:53,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:53,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:53,292 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:53,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:53,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:53,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:53,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:53,293 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:53,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:53,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:53,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:53,295 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:53,295 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:53,295 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:53,295 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:53,295 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:53,295 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:53,295 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:53,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:53,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:53,295 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:53,296 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:53,295 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:53,296 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:53,296 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:53,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:53,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:53,296 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:53,296 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:53,396 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:53,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:53,398 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:53,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:53,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:53,399 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:53,399 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:53,399 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:53,399 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:53,399 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:53,399 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:53,400 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:53,400 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:53,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:53,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:53,400 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:53,400 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:53,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:53,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:53,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:53,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:53,401 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:53,401 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:53,401 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:53,401 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:53,401 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:53,401 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:53,401 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:53,401 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:53,401 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:53,401 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:57:53,401 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e986ad248e3bbfc4197e05257d0f5eac 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-08 20:57:53,402 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=A 2023-04-08 20:57:53,402 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:53,402 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=B 2023-04-08 20:57:53,402 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:53,402 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=C 2023-04-08 20:57:53,402 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:53,419 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408fbcd3ffe174f4939a5103c68a8cfe782 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408fbcd3ffe174f4939a5103c68a8cfe782 2023-04-08 20:57:53,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:57:53,655 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:57:53,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680987473696, max timestamp=1680987473713Num uniques 859; 2023-04-08 20:57:53,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 20:57:53,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680987473696, max timestamp=1680987473713Num uniques 859; 2023-04-08 20:57:53,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 20:57:53,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680987473696, max timestamp=1680987473713Num uniques 859; 2023-04-08 20:57:53,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 20:57:53,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:53,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39370 deadline: 1680987533784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:53,823 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=154, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/A/268070a5d01144699275527c6e320e08 2023-04-08 20:57:53,891 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:53,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39370 deadline: 1680987533890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:54,096 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:54,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39370 deadline: 1680987534093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:54,230 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/B/d026f50e05c94cd2bb8800df671177c6 2023-04-08 20:57:54,232 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:54,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39348 deadline: 1680987534230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:54,295 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:54,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39324 deadline: 1680987534293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:54,296 DEBUG [Thread-3374] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8135 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac., hostname=jenkins-hbase20.apache.org,41091,1680987313601, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 20:57:54,400 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:54,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39370 deadline: 1680987534399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:54,639 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/C/3da986f783534e5792a4bc77536a5c77 2023-04-08 20:57:54,642 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/A/268070a5d01144699275527c6e320e08 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/268070a5d01144699275527c6e320e08 2023-04-08 20:57:54,644 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/268070a5d01144699275527c6e320e08, entries=150, sequenceid=154, filesize=25.3 K 2023-04-08 20:57:54,645 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/B/d026f50e05c94cd2bb8800df671177c6 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/d026f50e05c94cd2bb8800df671177c6 2023-04-08 20:57:54,647 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/d026f50e05c94cd2bb8800df671177c6, entries=150, sequenceid=154, filesize=11.8 K 2023-04-08 20:57:54,648 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/C/3da986f783534e5792a4bc77536a5c77 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/3da986f783534e5792a4bc77536a5c77 2023-04-08 20:57:54,651 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/3da986f783534e5792a4bc77536a5c77, entries=150, sequenceid=154, filesize=11.8 K 2023-04-08 20:57:54,652 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for e986ad248e3bbfc4197e05257d0f5eac in 1250ms, sequenceid=154, compaction requested=true 2023-04-08 20:57:54,652 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e986ad248e3bbfc4197e05257d0f5eac: 2023-04-08 20:57:54,652 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:54,652 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:54,652 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:54,652 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:54,652 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:54,652 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:54,653 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:54,653 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:54,653 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:54,653 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:54,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:54,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:54,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:54,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e7dbcbf[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:54,654 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:54,654 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:54,655 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:54,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:54,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:54,655 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:54,655 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:54,655 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:54,655 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:54,655 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:54,655 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:54,655 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:54,655 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:54,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:54,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:54,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:54,655 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:54,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:54,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:54,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:54,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:54,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:54,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:54,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:54,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:54,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:54,657 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:54,657 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:54,657 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:54,657 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:54,658 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:54,658 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:54,658 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:54,658 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:54,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:54,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:54,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:54,659 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:54,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:54,660 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:54,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:54,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:54,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:54,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:54,661 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:54,661 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:54,661 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:54,661 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:54,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:54,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:54,661 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:54,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:54,661 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:54,661 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:54,661 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:54,661 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:54,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:54,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:54,662 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:54,662 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:54,662 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:54,662 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:54,662 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:54,762 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:54,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:54,764 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:54,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:54,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:54,765 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:54,765 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:54,765 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:54,765 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:54,765 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:54,765 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:54,766 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:54,766 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:54,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:54,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:54,766 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:54,766 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:54,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:54,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:54,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:54,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:54,767 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:54,767 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:54,767 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:54,767 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:54,767 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:54,767 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:54,767 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:54,767 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:54,767 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:54,768 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:57:54,768 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e986ad248e3bbfc4197e05257d0f5eac 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-08 20:57:54,768 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=A 2023-04-08 20:57:54,768 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:57:54,768 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=B 2023-04-08 20:57:54,768 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:57:54,768 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=C 2023-04-08 20:57:54,768 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:57:54,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:57:54,906 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:57:54,935 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:54,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39370 deadline: 1680987534932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:55,036 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:55,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39370 deadline: 1680987535036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:55,185 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408a6436ff7b935411ca5e983095b7373ce to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408a6436ff7b935411ca5e983095b7373ce 2023-04-08 20:57:55,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:55,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39370 deadline: 1680987535238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:55,545 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:55,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39370 deadline: 1680987535542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:55,589 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=190, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/A/5a3505bca21f44198e22df7de45bcd27 2023-04-08 20:57:55,997 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/B/77d293431ac74e4aaf86ccf4cd14c249 2023-04-08 20:57:56,005 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/C/86187b52c12441e9a5cce74e48084aac 2023-04-08 20:57:56,008 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/A/5a3505bca21f44198e22df7de45bcd27 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/5a3505bca21f44198e22df7de45bcd27 2023-04-08 20:57:56,011 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/5a3505bca21f44198e22df7de45bcd27, entries=150, sequenceid=190, filesize=25.3 K 2023-04-08 20:57:56,012 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/B/77d293431ac74e4aaf86ccf4cd14c249 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/77d293431ac74e4aaf86ccf4cd14c249 2023-04-08 20:57:56,015 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/77d293431ac74e4aaf86ccf4cd14c249, entries=150, sequenceid=190, filesize=11.8 K 2023-04-08 20:57:56,015 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/C/86187b52c12441e9a5cce74e48084aac as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/86187b52c12441e9a5cce74e48084aac 2023-04-08 20:57:56,018 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/86187b52c12441e9a5cce74e48084aac, entries=150, sequenceid=190, filesize=11.8 K 2023-04-08 20:57:56,019 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for e986ad248e3bbfc4197e05257d0f5eac in 1250ms, sequenceid=190, compaction requested=true 2023-04-08 20:57:56,019 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e986ad248e3bbfc4197e05257d0f5eac: 2023-04-08 20:57:56,019 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:56,019 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:56,019 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:56,019 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,019 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:56,019 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:56,020 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,020 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:56,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:56,021 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,021 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:56,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:56,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:56,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:56,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:56,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d3e6cde[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:56,022 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:56,022 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,022 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,022 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:56,022 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:56,022 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,022 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:56,022 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:56,023 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:56,023 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:56,023 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,023 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:56,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:56,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:56,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:56,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:56,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:56,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:56,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:56,025 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:56,025 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:56,025 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:56,025 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:56,025 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:56,026 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,026 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,026 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:56,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:56,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:56,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:56,026 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:56,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:56,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:56,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,027 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:56,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,029 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:56,029 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:56,029 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:56,029 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:56,029 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,029 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,029 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,029 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:56,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:56,029 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:56,029 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,029 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:56,029 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:56,029 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,029 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:56,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:56,030 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:56,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:57:56,051 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e986ad248e3bbfc4197e05257d0f5eac 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-08 20:57:56,052 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=A 2023-04-08 20:57:56,052 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:56,052 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=B 2023-04-08 20:57:56,052 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:56,052 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=C 2023-04-08 20:57:56,052 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:56,117 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:56,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39370 deadline: 1680987536114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,129 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:56,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:56,131 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:56,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:56,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:56,132 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:56,132 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:56,132 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:56,132 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:56,133 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,133 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,133 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:56,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:56,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:56,133 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,134 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,134 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:56,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:56,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,134 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:56,134 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:56,134 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:56,134 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:56,134 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:56,134 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:56,135 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:56,135 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:56,135 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:56,135 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:57:56,135 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:57:56,135 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:56,135 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:56,135 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:56,135 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,135 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:56,135 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:56,136 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,136 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:56,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:56,136 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,136 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:56,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:56,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:56,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:56,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:56,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56b079e6[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:56,137 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:56,137 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,138 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,138 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:56,138 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:56,138 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:56,138 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,138 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:56,138 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:56,138 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:56,138 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,139 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:56,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:56,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:56,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:56,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:56,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:56,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:56,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:56,140 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:56,140 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:56,140 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:56,141 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:56,141 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:56,141 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,141 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:56,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,141 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:56,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:56,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:56,142 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:56,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:56,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:56,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,143 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:56,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,144 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,144 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,144 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,144 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:56,144 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:56,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:56,144 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:56,144 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,144 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:56,144 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,144 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:56,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:56,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:56,144 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,145 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:56,145 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:56,145 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:56,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:56,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39370 deadline: 1680987536218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,239 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:56,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39348 deadline: 1680987536237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,240 DEBUG [Thread-3370] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4133 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac., hostname=jenkins-hbase20.apache.org,41091,1680987313601, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 20:57:56,245 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:56,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:56,247 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:56,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:56,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:56,248 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:56,248 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:56,248 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:56,248 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:56,248 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,248 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,249 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:56,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:56,249 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:56,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,249 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,250 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:56,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:56,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,250 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:56,250 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:56,250 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:56,250 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:56,250 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:56,250 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:56,250 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:56,250 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:56,250 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:56,250 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:57:56,251 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:57:56,251 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:56,251 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:56,251 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:56,251 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,251 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:56,251 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:56,252 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,252 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:56,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:56,253 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,253 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:56,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:56,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:56,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:56,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:56,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50caf62e[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:56,254 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:56,254 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,255 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,255 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:56,255 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:56,255 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,255 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:56,255 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:56,255 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,255 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:56,255 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:56,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:56,256 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:56,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:56,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:56,257 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:56,257 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:56,257 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:56,257 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:56,257 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:56,258 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,258 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:56,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:56,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,258 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:56,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:56,259 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:56,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:56,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:56,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,260 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:56,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,261 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,261 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,261 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:56,261 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:56,261 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:56,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:56,261 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:56,261 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:56,261 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:56,261 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,261 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,261 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,261 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:56,261 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:56,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:56,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:56,261 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,361 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:56,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:56,363 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:56,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:56,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:56,364 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:56,364 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:56,364 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:56,364 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:56,365 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,365 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,365 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,365 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:56,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:56,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:56,366 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,366 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:56,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,366 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:56,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,366 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:56,367 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:56,367 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:56,367 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:56,367 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:56,367 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:56,367 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:56,367 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:56,367 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:57:56,367 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:57:56,367 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:56,367 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:56,367 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:56,367 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,367 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:56,367 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:56,368 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,368 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:56,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:56,368 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:56,368 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:56,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:56,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:56,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:56,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63e130cc[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:56,369 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:56,369 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,370 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,370 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:56,370 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:56,370 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:56,370 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,370 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:56,371 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:56,371 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,371 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:56,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:56,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:56,371 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:56,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:56,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:56,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:56,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:56,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:56,373 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:56,373 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:56,373 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:56,373 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:56,373 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:56,374 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,374 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:56,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,374 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:56,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:56,374 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:56,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:56,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:56,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:56,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,375 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:56,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,376 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,376 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,376 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,376 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,376 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,376 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,376 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:56,376 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:56,377 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:56,376 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,376 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:56,377 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:56,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:56,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:56,377 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:56,377 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:56,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:56,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:56,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:56,400 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:56,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39404 deadline: 1680987536397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,401 DEBUG [Thread-3368] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8178 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac., hostname=jenkins-hbase20.apache.org,41091,1680987313601, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 20:57:56,402 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:56,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39384 deadline: 1680987536399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,404 DEBUG [Thread-3366] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8180 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac., hostname=jenkins-hbase20.apache.org,41091,1680987313601, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 20:57:56,425 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:56,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39370 deadline: 1680987536423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,461 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040847fdd941d2c946b98c6e7be1a09126d2 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040847fdd941d2c946b98c6e7be1a09126d2 2023-04-08 20:57:56,477 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:56,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:56,479 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:56,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:56,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:56,480 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:56,480 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:56,480 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:56,480 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:56,480 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,480 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,481 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,481 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:56,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:56,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:56,481 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,481 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:56,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,482 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:56,482 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:56,482 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:56,482 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:56,482 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:56,482 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:56,482 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:56,482 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:56,482 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:56,483 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:57:56,483 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:57:56,483 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:56,483 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:56,483 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:56,483 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,483 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:56,483 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:56,483 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,483 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:56,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:56,484 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,484 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:56,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:56,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:56,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:56,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:56,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d57f8f2[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:56,485 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:56,485 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,485 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,485 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:56,485 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:56,485 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:56,486 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,486 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:56,486 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,486 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:56,486 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:56,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:56,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:56,486 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:56,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:56,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:56,488 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:56,488 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:56,488 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:56,488 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:56,488 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:56,488 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,488 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:56,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:56,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,488 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:56,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:56,489 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:56,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:56,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:56,490 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:56,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,491 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,491 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:56,491 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:56,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:56,491 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:56,491 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:56,491 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:56,491 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:56,491 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,492 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:56,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:56,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:56,492 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:56,492 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,492 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:56,492 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,492 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,592 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:56,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:56,593 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:56,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:56,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:56,595 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:56,595 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:56,595 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:56,595 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:56,595 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,595 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,596 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,596 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:56,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:56,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:56,596 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,596 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:56,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,598 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:56,598 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:56,598 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:56,598 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:56,598 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:56,598 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:56,598 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:56,599 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:56,599 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:56,599 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:57:56,599 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:57:56,599 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:56,599 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:56,599 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:56,599 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,599 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:56,599 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:56,600 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,600 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:56,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:56,600 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,600 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:56,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:56,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:56,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:56,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:56,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58fddff7[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:56,601 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:56,601 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,602 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,602 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:56,602 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:56,602 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,602 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:56,602 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:56,603 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:56,603 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,603 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:56,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:56,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:56,603 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:56,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:56,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:56,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:56,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:56,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:56,605 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:56,605 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:56,605 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:56,605 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:56,605 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:56,606 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,606 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:56,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:56,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,606 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:56,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:56,606 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:56,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:56,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:56,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,607 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:56,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,609 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,609 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,609 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:56,609 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:56,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:56,609 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:56,609 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:56,609 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,609 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,609 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,609 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:56,609 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:56,609 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:56,609 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:56,609 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:56,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:56,709 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:56,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:56,711 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:56,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:56,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:56,712 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:56,712 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:56,712 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:56,712 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:56,712 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,712 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,713 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,713 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:56,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:56,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:56,713 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,713 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:56,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:56,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,714 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:56,714 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:56,714 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:56,714 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:56,714 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:56,714 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:56,714 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:56,714 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:56,714 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:56,715 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:57:56,715 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:57:56,715 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:56,715 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:56,715 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:56,715 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,715 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:56,715 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:56,716 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:56,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:56,716 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:56,716 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,716 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:56,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:56,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:56,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:56,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f012940[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:56,717 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:56,717 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,718 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,718 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:56,718 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:56,718 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,718 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:56,718 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:56,719 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,719 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:56,719 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:56,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:56,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:56,719 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:56,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:56,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:56,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:56,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:56,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:56,721 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:56,721 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:56,721 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:56,721 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:56,721 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:56,722 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,722 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:56,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:56,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,722 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:56,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:56,722 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:56,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:56,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:56,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,723 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:56,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,724 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:56,724 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:56,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:56,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:56,724 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,725 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,725 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,725 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:56,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:56,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:56,725 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:56,725 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:56,725 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,725 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,725 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,725 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,725 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:56,725 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:56,725 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:56,728 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:56,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39370 deadline: 1680987536726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,825 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:56,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:56,827 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:56,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:56,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:56,828 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:56,828 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:56,828 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:56,828 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:56,828 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,828 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,829 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:56,829 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:56,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:56,829 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,829 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:56,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:56,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:56,830 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:56,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:56,830 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:56,830 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:56,830 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:56,830 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:56,830 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:56,830 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:56,830 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:57:56,830 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:57:56,831 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:56,831 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:56,831 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:56,831 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,831 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:56,831 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:56,831 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,831 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:56,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:56,832 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,832 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:56,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:56,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:56,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:56,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:56,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@350c053c[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:56,833 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:56,833 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,833 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,833 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:56,834 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,834 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:56,834 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:56,834 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:56,834 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:56,834 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:56,834 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:56,834 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:56,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:56,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:56,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:56,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:56,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:56,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:56,836 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:56,836 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:56,836 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:56,836 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:56,837 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:56,837 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,837 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,837 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:56,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:56,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:56,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,838 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:56,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:56,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:56,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:56,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,839 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:56,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,841 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,841 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:56,841 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,841 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:56,841 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:56,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:56,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:56,841 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,841 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:56,841 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:56,841 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:56,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:56,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:56,841 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:56,841 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,841 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,841 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,841 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,868 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=202, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/A/a24a0ca46d1d44a8a21f2d7f860e989c 2023-04-08 20:57:56,941 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:56,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:56,943 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:56,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:56,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:56,944 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:56,944 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:56,944 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:56,944 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:56,944 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,944 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,945 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:56,945 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:56,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:56,945 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,945 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:56,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:56,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,946 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:56,946 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:56,946 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:56,946 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:56,946 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:56,946 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:56,946 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:56,946 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:56,946 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:56,946 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:57:56,946 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:57:56,947 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:56,947 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:56,947 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:56,947 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,947 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:56,947 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:56,947 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:56,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:56,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:56,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:56,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,949 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:56,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:56,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7aeeb07[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:56,949 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,949 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:56,949 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:56,949 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,950 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,950 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:56,950 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,950 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:56,950 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:56,950 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:56,951 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:56,951 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:56,951 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,951 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:56,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:56,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:56,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:56,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:56,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:56,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:56,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:56,953 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:56,953 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:56,953 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:56,953 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:56,953 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:56,954 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,954 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:56,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:56,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,954 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:56,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:56,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:56,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:56,955 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:56,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,956 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:56,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:56,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,957 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,957 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,958 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:56,958 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:56,958 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:56,958 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,958 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:56,958 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:56,958 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:56,958 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:56,958 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:56,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:56,958 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:56,958 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:56,958 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:56,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:56,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:56,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:57,058 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:57,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:57,060 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:57,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:57,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:57,061 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:57,061 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:57,061 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:57,061 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:57,061 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,061 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:57,062 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,062 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:57,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:57,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:57,062 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,062 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:57,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:57,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:57,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:57,063 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:57,063 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:57,063 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:57,063 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:57,063 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:57,063 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:57,063 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:57,063 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:57,063 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:57,064 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:57:57,064 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:57:57,064 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:57,064 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:57,064 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:57,064 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,064 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:57,064 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:57,064 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,064 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:57,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:57,065 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,065 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:57,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:57,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:57,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:57,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:57,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:57,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@311a9e62[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:57,066 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:57,066 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,066 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,067 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:57,067 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,067 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:57,067 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:57,067 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:57,067 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,067 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:57,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:57,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:57,067 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:57,068 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:57,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:57,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:57,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:57,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:57,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:57,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:57,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:57,069 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:57,069 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:57,069 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:57,069 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:57,069 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:57,070 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,070 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,070 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:57,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:57,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:57,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:57,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:57,071 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:57,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:57,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:57,072 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:57,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:57,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,073 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,073 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:57,073 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:57,073 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,073 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:57,073 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,073 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:57,073 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:57,073 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:57,073 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,073 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,073 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:57,073 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:57,073 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:57,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:57,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:57,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:57,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:57,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:57,173 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:57,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:57,175 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:57,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:57,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:57,176 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:57,176 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:57,176 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:57,176 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:57,177 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,177 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:57,177 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,177 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:57,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:57,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:57,177 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,177 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:57,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:57,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:57,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:57,178 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:57,178 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:57,178 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:57,178 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:57,178 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:57,178 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:57,178 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:57,179 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:57,179 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:57,179 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:57:57,179 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:57:57,179 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:57,179 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:57,179 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:57,179 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,179 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:57,179 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:57,180 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,180 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:57,180 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,180 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:57,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:57,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6787fa28[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:57,181 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:57,181 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,181 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,181 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:57,181 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:57,181 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,182 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:57,181 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:57,182 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,182 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:57,182 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:57,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:57,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:57,182 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:57,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:57,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:57,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:57,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:57,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:57,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:57,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:57,184 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:57,184 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:57,184 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:57,184 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:57,184 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:57,185 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,185 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:57,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:57,185 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:57,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:57,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:57,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:57,185 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:57,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:57,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:57,186 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:57,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:57,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,187 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:57,187 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,187 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:57,187 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:57,187 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:57,187 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:57,187 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:57,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:57,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:57,187 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:57,188 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:57,188 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,188 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:57,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:57,188 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,188 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,188 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:57,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:57,233 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:57,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39370 deadline: 1680987537232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,276 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=202 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/B/62e0fa63ff704089a95502369af257bc 2023-04-08 20:57:57,288 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:57,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:57,289 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:57,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:57,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:57,290 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:57,290 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:57,290 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:57,290 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:57,291 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,291 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:57,291 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:57,291 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:57,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:57,292 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,292 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:57,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:57,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:57,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:57,292 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:57,293 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:57,293 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:57,293 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:57,293 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:57,293 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:57,293 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:57,293 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:57,293 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:57,293 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:57:57,293 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:57:57,293 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:57,294 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:57,294 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:57,294 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,294 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:57,294 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:57,295 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,295 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:57,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:57,295 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,295 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:57,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:57,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:57,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:57,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:57,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:57,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32772601[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:57,298 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:57,298 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,298 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,299 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:57,299 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:57,299 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:57,299 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,299 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:57,299 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:57,299 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:57,299 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,299 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:57,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:57,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:57,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:57,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:57,301 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:57,301 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:57,301 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:57,301 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:57,301 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:57,302 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,302 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,302 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:57,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:57,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:57,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:57,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:57,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:57,303 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:57,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:57,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:57,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,303 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,305 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,305 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:57,305 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:57,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:57,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:57,306 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:57,306 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:57,305 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:57,306 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:57,306 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:57,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:57,306 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:57,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:57,306 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,306 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:57,306 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,306 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,306 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,406 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:57,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:57,408 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:57,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:57,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:57,409 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:57,409 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:57,409 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:57,409 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:57,409 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,409 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:57,410 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:57,410 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:57,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:57,410 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,410 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:57,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:57,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:57,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:57,411 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:57,411 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:57,411 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:57,411 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:57,411 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:57,411 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:57,411 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:57,411 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:57,411 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:57,411 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:57:57,411 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:57:57,412 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:57,412 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:57,412 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:57,412 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,412 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:57,412 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:57,412 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,412 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:57,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:57,413 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,413 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:57,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:57,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:57,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:57,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:57,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:57,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4271059b[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:57,414 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:57,414 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,415 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,415 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:57,415 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:57,415 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,415 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:57,415 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:57,416 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:57,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:57,416 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:57,416 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:57,416 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:57,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:57,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:57,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:57,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:57,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:57,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:57,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:57,417 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:57,417 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:57,417 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:57,417 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:57,417 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:57,418 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,418 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,418 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:57,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:57,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:57,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:57,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:57,419 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:57,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:57,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:57,420 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:57,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:57,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:57,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,421 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,421 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:57,421 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:57,421 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:57,422 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:57,422 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:57,421 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:57,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:57,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:57,421 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:57,422 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:57,422 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:57,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:57,422 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:57,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:57,422 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,422 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,422 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,522 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:57,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:57,524 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:57,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:57,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:57,525 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:57,525 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:57,525 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:57,525 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:57,525 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,526 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:57,526 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,526 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:57,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:57,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:57,527 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,527 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:57,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:57,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:57,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:57,527 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:57,528 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:57,528 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:57,528 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:57,528 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:57,528 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:57,528 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:57,528 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:57,528 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:57,528 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:57:57,528 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:57:57,528 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:57,528 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:57,528 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:57,528 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,528 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:57,528 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:57,529 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,529 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:57,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:57,529 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,529 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:57,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:57,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:57,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:57,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:57,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:57,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@277d7a39[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:57,530 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:57,530 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,531 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,531 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:57,531 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,531 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:57,531 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:57,531 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:57,532 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,532 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:57,532 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:57,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:57,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:57,532 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:57,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:57,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:57,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:57,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:57,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:57,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:57,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:57,534 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:57,534 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:57,534 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:57,534 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:57,534 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:57,535 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,535 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,535 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:57,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:57,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:57,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:57,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:57,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:57,535 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:57,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:57,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:57,536 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:57,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:57,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,537 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:57,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:57,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:57,538 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,538 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:57,538 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:57,538 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:57,538 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:57,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:57,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:57,538 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:57,538 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:57,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:57,538 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:57,538 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:57,538 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,538 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,538 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,538 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,638 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:57,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:57,640 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:57,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:57,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:57,642 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:57,642 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:57,642 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:57,642 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:57,642 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,642 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:57,643 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:57,643 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:57,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:57,643 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,643 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:57,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:57,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:57,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:57,644 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:57,644 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:57,644 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:57,644 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:57,644 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:57,644 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:57,644 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:57,644 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:57,644 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:57,644 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:57:57,644 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:57:57,644 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:57,644 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:57,644 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:57,644 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,644 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:57,644 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:57,645 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,645 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:57,645 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,646 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:57,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:57,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:57,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:57,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:57,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ff39244[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:57,647 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:57,647 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,647 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,647 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:57,647 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:57,647 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:57,647 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,647 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:57,648 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,648 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:57,648 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:57,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:57,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:57,648 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:57,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:57,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:57,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:57,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:57,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:57,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:57,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:57,650 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:57,650 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:57,650 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:57,650 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:57,650 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:57,651 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,651 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,651 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:57,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:57,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:57,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:57,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:57,651 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:57,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:57,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:57,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:57,652 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:57,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:57,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,655 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,655 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:57,655 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:57,655 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:57,655 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:57,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:57,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:57,655 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:57,655 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:57,655 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:57,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:57:57,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:57,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:57,655 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:57,655 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,655 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,655 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:57,655 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,655 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:57,684 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=202 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/C/142ef174e9d34176809140bd7753fed1 2023-04-08 20:57:57,687 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/A/a24a0ca46d1d44a8a21f2d7f860e989c as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/a24a0ca46d1d44a8a21f2d7f860e989c 2023-04-08 20:57:57,690 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/a24a0ca46d1d44a8a21f2d7f860e989c, entries=200, sequenceid=202, filesize=32.1 K 2023-04-08 20:57:57,690 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/B/62e0fa63ff704089a95502369af257bc as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/62e0fa63ff704089a95502369af257bc 2023-04-08 20:57:57,693 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/62e0fa63ff704089a95502369af257bc, entries=150, sequenceid=202, filesize=11.8 K 2023-04-08 20:57:57,694 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/C/142ef174e9d34176809140bd7753fed1 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/142ef174e9d34176809140bd7753fed1 2023-04-08 20:57:57,696 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/142ef174e9d34176809140bd7753fed1, entries=150, sequenceid=202, filesize=11.8 K 2023-04-08 20:57:57,697 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for e986ad248e3bbfc4197e05257d0f5eac in 1646ms, sequenceid=202, compaction requested=true 2023-04-08 20:57:57,697 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e986ad248e3bbfc4197e05257d0f5eac: 2023-04-08 20:57:57,697 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:57:57,697 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-08 20:57:57,697 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-08 20:57:57,697 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:57:57,697 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:57:57,699 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 239304 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-08 20:57:57,699 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 107574 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-08 20:57:57,699 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): e986ad248e3bbfc4197e05257d0f5eac/A is initiating minor compaction (all files) 2023-04-08 20:57:57,699 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): e986ad248e3bbfc4197e05257d0f5eac/B is initiating minor compaction (all files) 2023-04-08 20:57:57,699 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e986ad248e3bbfc4197e05257d0f5eac/A in TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:57,699 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e986ad248e3bbfc4197e05257d0f5eac/B in TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:57,699 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/8c7834a3e2b0497fb62833f485237746, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/39fbfc5b9f1d45db951c704a49236afe, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/b7a7fae04c134dcab05eeacc02a6266a, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/1df5438db2c84d41a1bde13f7f90d8b6, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/b41833a47369456cb3bacb40b070d410, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/89ef2fd6787b4922a37291f8539ca12c, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/268070a5d01144699275527c6e320e08, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/5a3505bca21f44198e22df7de45bcd27, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/a24a0ca46d1d44a8a21f2d7f860e989c] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp, totalSize=233.7 K 2023-04-08 20:57:57,699 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/72dc29331ded40d08416a8a884400312, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/cd573042af1842a7a9c4a5fb3e5bc6fc, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/0de7c8d866c041209911a04fe1c18192, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/7fab73ecc11f44e8a2fba15761dede3c, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/7c91e95173484b28886611bdee7822fb, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/fdde235952c64d7193f5e601145b538b, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/d026f50e05c94cd2bb8800df671177c6, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/77d293431ac74e4aaf86ccf4cd14c249, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/62e0fa63ff704089a95502369af257bc] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp, totalSize=105.1 K 2023-04-08 20:57:57,699 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 8c7834a3e2b0497fb62833f485237746, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680987463980 2023-04-08 20:57:57,699 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 72dc29331ded40d08416a8a884400312, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680987463980 2023-04-08 20:57:57,700 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 39fbfc5b9f1d45db951c704a49236afe, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680987464028 2023-04-08 20:57:57,700 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting cd573042af1842a7a9c4a5fb3e5bc6fc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680987464028 2023-04-08 20:57:57,700 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting b7a7fae04c134dcab05eeacc02a6266a, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680987466150 2023-04-08 20:57:57,700 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 0de7c8d866c041209911a04fe1c18192, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680987466150 2023-04-08 20:57:57,700 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 1df5438db2c84d41a1bde13f7f90d8b6, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1680987468217 2023-04-08 20:57:57,700 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 7fab73ecc11f44e8a2fba15761dede3c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1680987468217 2023-04-08 20:57:57,700 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting b41833a47369456cb3bacb40b070d410, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680987469314 2023-04-08 20:57:57,700 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 7c91e95173484b28886611bdee7822fb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680987469314 2023-04-08 20:57:57,700 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 89ef2fd6787b4922a37291f8539ca12c, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1680987471509 2023-04-08 20:57:57,701 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting fdde235952c64d7193f5e601145b538b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1680987471509 2023-04-08 20:57:57,701 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 268070a5d01144699275527c6e320e08, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1680987472090 2023-04-08 20:57:57,701 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting d026f50e05c94cd2bb8800df671177c6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1680987472090 2023-04-08 20:57:57,701 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 5a3505bca21f44198e22df7de45bcd27, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1680987473769 2023-04-08 20:57:57,701 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 77d293431ac74e4aaf86ccf4cd14c249, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1680987473769 2023-04-08 20:57:57,701 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting a24a0ca46d1d44a8a21f2d7f860e989c, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1680987474916 2023-04-08 20:57:57,701 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 62e0fa63ff704089a95502369af257bc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1680987474916 2023-04-08 20:57:57,714 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): e986ad248e3bbfc4197e05257d0f5eac#B#compaction#473 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:57:57,716 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): e986ad248e3bbfc4197e05257d0f5eac#A#compaction#472 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:57:57,755 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:57,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:57,757 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:57,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:57,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:57,758 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:57,758 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:57,758 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:57,758 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:57,759 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,759 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:57,759 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:57,759 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:57,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:57,759 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:57,760 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:57,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:57,760 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:57,760 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:57,760 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:57,760 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:57,760 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:57,760 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:57,760 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:57,760 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:57,760 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:57,760 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:57:57,761 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e986ad248e3bbfc4197e05257d0f5eac 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-08 20:57:57,761 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=A 2023-04-08 20:57:57,761 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:57,761 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=B 2023-04-08 20:57:57,761 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:57,761 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=C 2023-04-08 20:57:57,761 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:58,125 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/B/dd47585719bf491f8f06d499920c549b as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/dd47585719bf491f8f06d499920c549b 2023-04-08 20:57:58,131 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in e986ad248e3bbfc4197e05257d0f5eac/B of e986ad248e3bbfc4197e05257d0f5eac into dd47585719bf491f8f06d499920c549b(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:57:58,131 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e986ad248e3bbfc4197e05257d0f5eac: 2023-04-08 20:57:58,131 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac., storeName=e986ad248e3bbfc4197e05257d0f5eac/B, priority=7, startTime=1680987477697; duration=0sec 2023-04-08 20:57:58,131 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:57:58,131 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-08 20:57:58,133 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 107574 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-08 20:57:58,133 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): e986ad248e3bbfc4197e05257d0f5eac/C is initiating minor compaction (all files) 2023-04-08 20:57:58,133 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e986ad248e3bbfc4197e05257d0f5eac/C in TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:58,134 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/e65543d837f940a49870fed5e1ed360b, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/1837a5958ab34dcd8b72ab59fcc44e27, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/d7cca22e678a4c828c1d204a48518955, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/ecc7240eb5264597bbe3a85160fab2cf, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/7329a7d4e110480e85f639e9714bcb88, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/fafdf2a87caf4e2588ef70fcff4855ee, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/3da986f783534e5792a4bc77536a5c77, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/86187b52c12441e9a5cce74e48084aac, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/142ef174e9d34176809140bd7753fed1] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp, totalSize=105.1 K 2023-04-08 20:57:58,134 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting e65543d837f940a49870fed5e1ed360b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680987463980 2023-04-08 20:57:58,134 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 1837a5958ab34dcd8b72ab59fcc44e27, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680987464028 2023-04-08 20:57:58,134 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting d7cca22e678a4c828c1d204a48518955, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680987466150 2023-04-08 20:57:58,134 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting ecc7240eb5264597bbe3a85160fab2cf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1680987468217 2023-04-08 20:57:58,135 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 7329a7d4e110480e85f639e9714bcb88, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680987469314 2023-04-08 20:57:58,135 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting fafdf2a87caf4e2588ef70fcff4855ee, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1680987471509 2023-04-08 20:57:58,135 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 3da986f783534e5792a4bc77536a5c77, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1680987472090 2023-04-08 20:57:58,135 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 86187b52c12441e9a5cce74e48084aac, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1680987473769 2023-04-08 20:57:58,135 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 142ef174e9d34176809140bd7753fed1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1680987474916 2023-04-08 20:57:58,144 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): e986ad248e3bbfc4197e05257d0f5eac#C#compaction#475 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:57:58,169 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304085c0ddc9218b44f13a40779cfad361fb8 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304085c0ddc9218b44f13a40779cfad361fb8 2023-04-08 20:57:58,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:57:58,241 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:57:58,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:58,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39370 deadline: 1680987538276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:58,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:58,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39370 deadline: 1680987538381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:58,532 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/A/877b1aa980544e52908cd707237cbc75 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/877b1aa980544e52908cd707237cbc75 2023-04-08 20:57:58,536 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in e986ad248e3bbfc4197e05257d0f5eac/A of e986ad248e3bbfc4197e05257d0f5eac into 877b1aa980544e52908cd707237cbc75(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:57:58,536 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e986ad248e3bbfc4197e05257d0f5eac: 2023-04-08 20:57:58,536 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac., storeName=e986ad248e3bbfc4197e05257d0f5eac/A, priority=7, startTime=1680987477697; duration=0sec 2023-04-08 20:57:58,536 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:57:58,552 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/C/e260fed857474dc8b19f37a7df80b5b3 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/e260fed857474dc8b19f37a7df80b5b3 2023-04-08 20:57:58,556 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in e986ad248e3bbfc4197e05257d0f5eac/C of e986ad248e3bbfc4197e05257d0f5eac into e260fed857474dc8b19f37a7df80b5b3(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:57:58,556 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e986ad248e3bbfc4197e05257d0f5eac: 2023-04-08 20:57:58,556 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac., storeName=e986ad248e3bbfc4197e05257d0f5eac/C, priority=7, startTime=1680987477697; duration=0sec 2023-04-08 20:57:58,556 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:57:58,573 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=222, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/A/5652a04eba2b4cc78f4b221e346bd438 2023-04-08 20:57:58,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:58,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39370 deadline: 1680987538583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:58,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:58,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39370 deadline: 1680987538885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:58,981 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=222 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/B/543ced48dc984fc7a3e7ee896a777710 2023-04-08 20:57:58,988 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=222 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/C/668ee8fba45a4a37a1161a03dd3b2204 2023-04-08 20:57:58,991 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/A/5652a04eba2b4cc78f4b221e346bd438 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/5652a04eba2b4cc78f4b221e346bd438 2023-04-08 20:57:58,993 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/5652a04eba2b4cc78f4b221e346bd438, entries=150, sequenceid=222, filesize=25.3 K 2023-04-08 20:57:58,994 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/B/543ced48dc984fc7a3e7ee896a777710 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/543ced48dc984fc7a3e7ee896a777710 2023-04-08 20:57:58,997 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/543ced48dc984fc7a3e7ee896a777710, entries=150, sequenceid=222, filesize=11.8 K 2023-04-08 20:57:58,997 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/C/668ee8fba45a4a37a1161a03dd3b2204 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/668ee8fba45a4a37a1161a03dd3b2204 2023-04-08 20:57:59,000 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/668ee8fba45a4a37a1161a03dd3b2204, entries=150, sequenceid=222, filesize=11.8 K 2023-04-08 20:57:59,001 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for e986ad248e3bbfc4197e05257d0f5eac in 1241ms, sequenceid=222, compaction requested=false 2023-04-08 20:57:59,001 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e986ad248e3bbfc4197e05257d0f5eac: 2023-04-08 20:57:59,001 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:59,001 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:59,001 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:59,001 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:59,001 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:59,001 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:59,002 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:59,002 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:59,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:59,003 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:59,003 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:57:59,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:59,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:57:59,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@743fe495[Count = 0] remaining members to acquire global barrier 2023-04-08 20:57:59,007 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:57:59,007 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:59,008 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:59,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:59,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:59,008 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:57:59,008 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:57:59,008 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:57:59,008 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:59,008 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:57:59,009 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:59,009 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:57:59,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:59,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:59,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:59,009 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:59,009 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:57:59,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:59,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:59,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:59,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:59,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:57:59,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:57:59,011 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:57:59,011 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:57:59,011 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:57:59,011 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:57:59,011 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:57:59,012 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:59,012 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:59,012 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:59,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:59,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:59,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:59,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:57:59,013 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:59,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:59,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:59,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:57:59,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:59,014 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:57:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:57:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:59,015 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:59,016 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:59,016 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:59,016 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:59,016 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:57:59,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:59,016 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:59,016 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:59,016 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:57:59,016 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:59,016 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:57:59,016 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:57:59,016 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:57:59,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:57:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:57:59,016 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:57:59,016 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:57:59,116 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:57:59,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:57:59,118 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:57:59,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:57:59,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:57:59,119 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:57:59,119 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:59,119 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:57:59,119 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:57:59,119 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:59,119 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:59,120 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:59,120 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:57:59,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:57:59,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:57:59,120 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:59,120 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:57:59,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:59,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:57:59,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:57:59,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:57:59,121 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:57:59,121 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:57:59,121 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:57:59,121 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:57:59,121 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:57:59,121 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:57:59,121 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:57:59,121 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:57:59,121 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:59,121 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:57:59,121 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e986ad248e3bbfc4197e05257d0f5eac 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-08 20:57:59,121 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=A 2023-04-08 20:57:59,121 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:59,121 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=B 2023-04-08 20:57:59,121 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:59,122 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=C 2023-04-08 20:57:59,122 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:57:59,144 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304080fd4d5f9e8104df39105ee26107c293f to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304080fd4d5f9e8104df39105ee26107c293f 2023-04-08 20:57:59,148 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=237, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/A/628561125e744bfaa7903cb06443a8d2 2023-04-08 20:57:59,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:57:59,391 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:57:59,427 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:59,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39370 deadline: 1680987539425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:59,529 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:59,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39370 deadline: 1680987539527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:59,575 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=237 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/B/b37d219dc04140ea87829664ce4a8a8c 2023-04-08 20:57:59,731 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:57:59,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39370 deadline: 1680987539730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:59,983 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=237 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/C/c38d5e7d7c1f441c8141f6ce475fcefd 2023-04-08 20:57:59,987 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/A/628561125e744bfaa7903cb06443a8d2 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/628561125e744bfaa7903cb06443a8d2 2023-04-08 20:57:59,990 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/628561125e744bfaa7903cb06443a8d2, entries=150, sequenceid=237, filesize=25.3 K 2023-04-08 20:57:59,991 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/B/b37d219dc04140ea87829664ce4a8a8c as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/b37d219dc04140ea87829664ce4a8a8c 2023-04-08 20:57:59,994 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/b37d219dc04140ea87829664ce4a8a8c, entries=150, sequenceid=237, filesize=11.8 K 2023-04-08 20:57:59,995 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/C/c38d5e7d7c1f441c8141f6ce475fcefd as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/c38d5e7d7c1f441c8141f6ce475fcefd 2023-04-08 20:57:59,997 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/c38d5e7d7c1f441c8141f6ce475fcefd, entries=150, sequenceid=237, filesize=11.8 K 2023-04-08 20:57:59,998 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for e986ad248e3bbfc4197e05257d0f5eac in 877ms, sequenceid=237, compaction requested=true 2023-04-08 20:57:59,998 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e986ad248e3bbfc4197e05257d0f5eac: 2023-04-08 20:57:59,998 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:57:59,998 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:57:59,998 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:57:59,998 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:59,998 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:57:59,998 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:57:59,999 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:59,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:57:59,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:57:59,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:57:59,999 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:00,000 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,000 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:58:00,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:00,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:00,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:58:00,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e286a54[Count = 0] remaining members to acquire global barrier 2023-04-08 20:58:00,001 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:58:00,001 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,001 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,002 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:58:00,002 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,002 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:58:00,002 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:58:00,002 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:58:00,002 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,002 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:58:00,003 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:58:00,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:00,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:00,003 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:58:00,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:00,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:00,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:00,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:58:00,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:58:00,005 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:58:00,005 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:58:00,005 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:58:00,005 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:58:00,005 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:58:00,006 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,006 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:00,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:00,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,006 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:58:00,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:58:00,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:58:00,006 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:00,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:00,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,007 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:00,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,009 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:58:00,009 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:58:00,009 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,009 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,009 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:58:00,009 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:58:00,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:58:00,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:58:00,009 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,009 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:58:00,009 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:58:00,009 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:58:00,009 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,009 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:58:00,009 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,010 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:58:00,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:58:00,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:58:00,037 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e986ad248e3bbfc4197e05257d0f5eac 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-08 20:58:00,037 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=A 2023-04-08 20:58:00,037 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:58:00,037 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=B 2023-04-08 20:58:00,037 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:58:00,037 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=C 2023-04-08 20:58:00,037 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:58:00,072 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:58:00,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39370 deadline: 1680987540071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,109 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:58:00,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:58:00,111 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:58:00,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:58:00,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:58:00,112 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:58:00,112 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:58:00,112 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:58:00,112 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:58:00,112 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,113 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,113 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:58:00,113 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:58:00,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:58:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,114 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,114 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:58:00,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:58:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,114 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:58:00,114 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:58:00,115 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:58:00,115 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:58:00,115 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:58:00,116 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:58:00,116 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:58:00,116 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:00,116 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:58:00,116 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:58:00,116 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:00,116 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:58:00,116 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:58:00,116 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,116 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:58:00,116 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:58:00,117 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,117 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:00,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:00,117 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,117 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:58:00,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:00,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:00,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:00,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:58:00,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7248731a[Count = 0] remaining members to acquire global barrier 2023-04-08 20:58:00,119 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:58:00,119 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,119 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,120 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:58:00,120 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:58:00,120 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:58:00,120 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,120 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:58:00,120 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,120 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:58:00,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:00,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:00,120 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:58:00,120 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:58:00,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:00,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:00,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:00,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:58:00,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:58:00,122 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:58:00,122 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:58:00,123 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:58:00,123 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:58:00,123 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:58:00,123 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,123 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:00,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:00,123 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:58:00,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:00,124 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:58:00,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:58:00,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:00,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,125 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:00,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,126 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,126 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,126 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,126 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:58:00,126 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:58:00,126 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,126 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:58:00,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:58:00,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:58:00,126 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:58:00,127 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:58:00,126 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,127 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,127 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,127 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:58:00,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:58:00,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:58:00,127 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:58:00,176 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:58:00,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39370 deadline: 1680987540173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,227 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:58:00,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:58:00,229 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:58:00,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:58:00,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:58:00,230 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:58:00,230 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:58:00,230 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:58:00,230 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:58:00,230 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,230 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,232 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,232 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:58:00,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:58:00,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:58:00,232 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,232 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:58:00,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:58:00,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,233 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:58:00,233 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:58:00,233 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:58:00,233 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:58:00,233 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:58:00,233 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:58:00,233 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:58:00,233 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:58:00,234 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:00,234 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:58:00,234 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:58:00,234 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:00,234 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:58:00,234 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:58:00,234 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,234 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:58:00,234 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:58:00,235 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,235 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:00,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:00,235 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,235 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:58:00,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:00,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:00,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:00,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:58:00,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@114abea5[Count = 0] remaining members to acquire global barrier 2023-04-08 20:58:00,236 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:58:00,236 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,237 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,237 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:58:00,237 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,237 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:58:00,237 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:58:00,237 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:58:00,238 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,238 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:58:00,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:00,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:00,238 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:58:00,238 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:58:00,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:00,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:00,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:00,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:58:00,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:58:00,240 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:58:00,240 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:58:00,240 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:58:00,240 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:58:00,240 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:58:00,241 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,241 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:00,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:00,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,241 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:58:00,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:00,241 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:58:00,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:58:00,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:00,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,242 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:00,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,243 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,243 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,243 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:58:00,243 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:58:00,243 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:58:00,244 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:58:00,243 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,243 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:58:00,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:58:00,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:58:00,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:58:00,244 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,244 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:58:00,244 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,244 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:58:00,244 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:58:00,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:58:00,244 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,266 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:58:00,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39348 deadline: 1680987540265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,267 DEBUG [Thread-3370] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8160 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac., hostname=jenkins-hbase20.apache.org,41091,1680987313601, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 20:58:00,344 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:58:00,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:58:00,346 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:58:00,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:58:00,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:58:00,347 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:58:00,347 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:58:00,347 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:58:00,347 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:58:00,347 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,347 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,348 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,348 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:58:00,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:58:00,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:58:00,348 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,348 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:58:00,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:58:00,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,349 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:58:00,349 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:58:00,349 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:58:00,349 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:58:00,349 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:58:00,349 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:58:00,349 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:58:00,349 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:58:00,349 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:00,349 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:58:00,349 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:58:00,349 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:00,350 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:58:00,350 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:58:00,350 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,350 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:58:00,350 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:58:00,350 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,350 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:00,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:00,351 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,351 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:58:00,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:00,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:00,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:00,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:58:00,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1adafb55[Count = 0] remaining members to acquire global barrier 2023-04-08 20:58:00,352 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:58:00,352 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,352 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,353 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:58:00,353 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:58:00,353 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,353 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:58:00,353 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:58:00,353 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,353 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:58:00,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:00,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:00,353 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:58:00,354 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:58:00,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:00,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:00,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:00,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:58:00,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:58:00,356 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:58:00,356 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:58:00,356 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:58:00,356 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:58:00,356 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:58:00,357 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:00,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:00,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:00,357 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,357 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,357 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:58:00,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:58:00,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:58:00,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:00,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,358 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:00,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,359 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,359 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,359 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,359 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,359 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,359 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,359 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:58:00,359 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:58:00,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:58:00,360 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:58:00,360 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:58:00,360 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:58:00,360 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:58:00,360 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:58:00,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:58:00,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:58:00,360 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:58:00,379 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:58:00,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39370 deadline: 1680987540378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,455 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040855067c13e5f64ef8a0394314065d3aaa to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040855067c13e5f64ef8a0394314065d3aaa 2023-04-08 20:58:00,460 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:58:00,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:58:00,462 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:58:00,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:58:00,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:58:00,463 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:58:00,463 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:58:00,463 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:58:00,463 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:58:00,463 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,463 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,464 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,464 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:58:00,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:58:00,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:58:00,464 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,464 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:58:00,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:58:00,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,465 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:58:00,465 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:58:00,465 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:58:00,465 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:58:00,465 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:58:00,465 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:58:00,465 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:58:00,465 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:58:00,465 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:00,466 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:58:00,466 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:58:00,466 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:00,466 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:58:00,466 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:58:00,466 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,466 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:58:00,466 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:58:00,466 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:00,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:00,467 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:00,467 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,467 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:58:00,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:00,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:00,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:58:00,468 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:58:00,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67dd4890[Count = 0] remaining members to acquire global barrier 2023-04-08 20:58:00,468 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,468 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,469 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:58:00,469 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:58:00,469 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:58:00,469 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,469 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:58:00,470 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:58:00,470 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,470 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:58:00,470 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:58:00,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:00,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:00,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:00,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:00,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:00,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:58:00,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:58:00,473 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:58:00,473 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:58:00,473 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:58:00,473 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:58:00,473 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:58:00,474 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,474 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:00,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:00,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,474 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:58:00,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:00,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:58:00,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:58:00,474 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:00,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,475 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:00,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,476 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,476 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:58:00,476 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:58:00,476 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,477 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,477 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:58:00,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:58:00,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:58:00,477 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:58:00,477 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:58:00,477 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,477 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,477 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,477 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:58:00,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:58:00,477 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:58:00,477 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:58:00,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:58:00,577 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:58:00,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:58:00,579 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:58:00,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:58:00,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:58:00,580 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:58:00,580 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:58:00,580 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:58:00,580 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:58:00,580 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,580 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,581 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,581 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:58:00,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:58:00,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:58:00,581 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,581 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:58:00,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:58:00,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,582 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:58:00,582 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:58:00,582 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:58:00,582 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:58:00,582 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:58:00,582 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:58:00,582 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:58:00,582 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:58:00,582 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:00,582 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:58:00,583 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:58:00,583 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:00,583 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:58:00,583 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:58:00,583 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,583 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:58:00,583 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:58:00,583 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,584 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:00,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:00,584 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,584 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:58:00,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:00,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:00,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:00,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:58:00,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56bc37d2[Count = 0] remaining members to acquire global barrier 2023-04-08 20:58:00,585 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:58:00,585 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,586 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,586 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:58:00,586 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:58:00,586 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,586 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:58:00,586 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:58:00,587 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,587 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:58:00,587 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:58:00,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:00,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:00,587 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:58:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:00,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:58:00,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:58:00,589 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:58:00,589 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:58:00,589 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:58:00,589 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:58:00,589 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:58:00,590 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,590 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:00,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:00,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,590 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:58:00,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:00,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:58:00,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:58:00,591 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:00,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,592 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,594 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,594 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,594 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:58:00,594 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:58:00,594 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:58:00,594 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:58:00,594 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:58:00,594 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:58:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:58:00,594 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,594 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,594 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,594 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:58:00,594 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,594 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:58:00,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:58:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:58:00,683 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:58:00,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39370 deadline: 1680987540682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,694 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:58:00,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:58:00,696 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:58:00,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:58:00,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:58:00,697 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:58:00,697 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:58:00,697 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:58:00,697 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:58:00,697 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,697 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,698 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,698 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:58:00,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:58:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:58:00,698 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,698 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:58:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:58:00,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,699 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:58:00,699 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:58:00,699 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:58:00,699 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:58:00,699 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:58:00,699 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:58:00,699 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:58:00,699 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:58:00,699 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:00,699 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:58:00,699 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:58:00,699 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:00,699 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:58:00,699 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:58:00,700 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,700 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:58:00,700 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:58:00,700 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,700 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:00,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:00,700 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,701 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:58:00,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:00,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:00,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:00,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:58:00,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12c70436[Count = 0] remaining members to acquire global barrier 2023-04-08 20:58:00,702 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:58:00,702 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,703 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,703 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:58:00,703 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,703 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:58:00,703 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:58:00,703 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:58:00,704 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:00,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:00,704 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:58:00,704 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:58:00,704 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:58:00,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:00,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:00,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:00,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:58:00,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:58:00,705 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:58:00,705 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:58:00,705 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:58:00,705 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:58:00,705 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:58:00,706 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,706 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:00,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,706 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:58:00,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:00,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:58:00,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:58:00,707 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:00,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,707 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:00,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,709 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,709 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:58:00,709 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,709 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:58:00,709 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,709 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,709 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:58:00,709 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:58:00,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:58:00,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:58:00,709 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,709 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,709 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,709 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:58:00,709 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:58:00,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:58:00,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:58:00,709 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:58:00,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:58:00,809 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:58:00,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:58:00,811 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:58:00,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:58:00,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:58:00,812 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:58:00,812 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:58:00,812 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:58:00,812 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:58:00,813 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,813 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,813 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:58:00,813 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:58:00,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:58:00,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,814 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,814 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:58:00,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:58:00,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,814 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:58:00,814 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:58:00,814 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:58:00,814 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:58:00,814 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:58:00,814 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:58:00,815 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:58:00,815 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:58:00,815 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:00,815 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:58:00,815 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:58:00,815 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:00,815 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:58:00,815 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:58:00,815 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,815 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:58:00,815 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:58:00,816 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,816 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:00,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:00,816 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,816 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:58:00,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:00,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:00,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:00,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:58:00,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f0a5568[Count = 0] remaining members to acquire global barrier 2023-04-08 20:58:00,817 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:58:00,817 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,818 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,818 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:58:00,818 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:58:00,818 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:58:00,818 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,818 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:58:00,819 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,819 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:58:00,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:00,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:00,819 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:58:00,819 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:58:00,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:00,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:00,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:00,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:58:00,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:58:00,821 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:58:00,821 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:58:00,821 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:58:00,821 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:58:00,821 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:58:00,822 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,822 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:00,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:00,822 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:58:00,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:00,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:58:00,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:58:00,822 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:00,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,823 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:00,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,825 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:58:00,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:58:00,825 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:58:00,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:58:00,825 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:58:00,825 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:58:00,825 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:58:00,825 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,825 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:58:00,825 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:58:00,825 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:58:00,825 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:58:00,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:58:00,825 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,825 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,825 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,825 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,860 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=258, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/A/d17e0b2f780a4d34bc507a2534e6145e 2023-04-08 20:58:00,925 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:58:00,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:58:00,927 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:58:00,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:58:00,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:58:00,928 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:58:00,928 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:58:00,928 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:58:00,928 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:58:00,929 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,929 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,929 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:58:00,929 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:58:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:58:00,930 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,930 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:58:00,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:58:00,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,930 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:58:00,930 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:58:00,930 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:58:00,930 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:58:00,930 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:58:00,931 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:58:00,931 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:58:00,931 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:58:00,931 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:00,931 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:58:00,931 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:58:00,931 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:00,931 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:58:00,931 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:58:00,931 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,931 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:58:00,931 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:58:00,932 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,932 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:00,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:00,932 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,932 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:58:00,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:00,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:00,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:00,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:58:00,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@521e1fd1[Count = 0] remaining members to acquire global barrier 2023-04-08 20:58:00,933 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:58:00,933 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,934 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,934 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:58:00,934 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,934 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:58:00,934 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:58:00,934 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:58:00,935 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,935 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:58:00,935 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:58:00,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:00,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:00,935 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:58:00,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:00,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:00,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:00,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:58:00,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:58:00,937 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:58:00,937 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:58:00,937 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:58:00,937 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:58:00,937 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:58:00,938 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,938 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,938 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:58:00,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:00,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:00,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:58:00,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:58:00,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:00,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:00,939 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:00,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,940 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:00,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:00,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,941 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:58:00,941 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:58:00,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:58:00,941 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:58:00,941 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:58:00,941 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,941 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:58:00,941 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:58:00,941 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:00,941 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:00,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:58:00,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:58:00,941 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,942 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:58:00,942 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:00,942 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:58:00,942 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,042 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:58:01,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:58:01,044 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:58:01,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:58:01,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:58:01,044 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:58:01,044 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:58:01,044 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:58:01,044 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:58:01,045 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,045 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:01,045 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,045 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:58:01,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:58:01,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:58:01,046 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,046 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:58:01,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:01,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:58:01,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:01,046 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:58:01,046 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:58:01,046 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:58:01,046 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:58:01,046 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:58:01,047 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:58:01,047 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:58:01,047 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:58:01,047 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:01,047 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:58:01,047 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:58:01,047 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:01,047 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:58:01,047 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:58:01,047 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,047 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:58:01,047 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:58:01,048 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:01,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:01,048 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:01,048 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,048 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:58:01,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:01,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:01,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:01,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:58:01,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@652bb0e7[Count = 0] remaining members to acquire global barrier 2023-04-08 20:58:01,050 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:58:01,050 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,050 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,050 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:58:01,050 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:58:01,050 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,050 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:58:01,050 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:58:01,051 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,051 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:58:01,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:01,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:01,051 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:58:01,051 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:58:01,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:01,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:01,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:01,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:01,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:01,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:58:01,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:58:01,053 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:58:01,053 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:58:01,053 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:58:01,053 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:58:01,053 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:58:01,054 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,054 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:01,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:01,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,054 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:58:01,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:01,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:58:01,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:01,054 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:58:01,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:01,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:01,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:01,055 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:01,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,057 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,057 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:58:01,057 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:01,057 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:58:01,057 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:58:01,057 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:01,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:58:01,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:58:01,057 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:58:01,057 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:58:01,057 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:58:01,057 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:58:01,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:58:01,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:58:01,057 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,057 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,058 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,058 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,157 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:58:01,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:58:01,159 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:58:01,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:58:01,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:58:01,160 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:58:01,160 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:58:01,160 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:58:01,160 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:58:01,161 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,161 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:01,161 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:58:01,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:58:01,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:58:01,161 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:01,162 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,162 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:58:01,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:58:01,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:01,162 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:58:01,162 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:58:01,163 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:58:01,163 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:58:01,163 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:58:01,163 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:58:01,163 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:58:01,163 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:58:01,163 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:01,163 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:58:01,163 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:58:01,163 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:01,163 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:58:01,163 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:58:01,163 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,163 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:58:01,163 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:58:01,164 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,164 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:01,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:01,164 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,164 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:58:01,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:01,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:01,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:01,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:01,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:58:01,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fe7eec0[Count = 0] remaining members to acquire global barrier 2023-04-08 20:58:01,166 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:58:01,166 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,167 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,167 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:58:01,167 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:58:01,167 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:58:01,167 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,167 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:58:01,168 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:58:01,168 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:58:01,168 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:01,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:01,168 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:58:01,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:01,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:01,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:01,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:01,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:01,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:58:01,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:58:01,170 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:58:01,170 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:58:01,170 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:58:01,170 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:58:01,170 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:58:01,171 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,171 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:58:01,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,171 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:01,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:01,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:58:01,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:58:01,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:01,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,172 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:01,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:01,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:01,173 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:01,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:01,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,175 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,175 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:01,175 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:01,175 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:58:01,175 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,175 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,175 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,175 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:58:01,175 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:58:01,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:58:01,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:58:01,175 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:58:01,175 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:58:01,175 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:58:01,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:58:01,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:58:01,176 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:58:01,176 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:58:01,186 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:58:01,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=41091] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39370 deadline: 1680987541184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,268 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=258 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/B/6c3201972b0148b1b3aa094050d91198 2023-04-08 20:58:01,276 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:58:01,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:58:01,277 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:58:01,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:58:01,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:58:01,281 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:58:01,281 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:58:01,281 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:58:01,281 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:58:01,281 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,281 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:01,282 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,282 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:58:01,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:58:01,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:58:01,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:01,282 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,283 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:58:01,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:58:01,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:01,283 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:58:01,286 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:58:01,286 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:58:01,286 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:58:01,286 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:58:01,286 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:58:01,288 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:58:01,288 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:58:01,288 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:01,288 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:58:01,288 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:58:01,288 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:01,288 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:58:01,289 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:58:01,289 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,289 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:58:01,289 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:58:01,297 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,297 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:01,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:01,297 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,297 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:58:01,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:01,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:01,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:01,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:01,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:58:01,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42cb5341[Count = 0] remaining members to acquire global barrier 2023-04-08 20:58:01,298 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:58:01,298 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,299 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,299 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:58:01,299 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:58:01,299 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:58:01,299 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,300 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:58:01,300 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:01,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:01,300 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:58:01,300 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:58:01,300 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:58:01,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:01,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:01,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:01,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:01,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:01,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:58:01,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:58:01,311 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:58:01,311 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:58:01,311 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:58:01,311 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:58:01,311 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:58:01,313 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,313 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,313 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:58:01,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:01,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:01,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:58:01,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:58:01,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:01,314 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:01,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:01,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:01,315 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:01,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:01,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,317 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,317 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:01,317 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:01,317 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,317 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:58:01,317 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:58:01,317 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:58:01,317 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:58:01,317 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,317 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:58:01,318 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:58:01,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:58:01,318 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:58:01,318 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:58:01,318 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:58:01,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:58:01,418 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:58:01,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:58:01,419 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:58:01,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:58:01,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:58:01,420 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:58:01,420 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:58:01,420 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:58:01,420 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:58:01,421 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,421 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:01,421 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,421 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:58:01,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:58:01,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:58:01,422 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,422 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:58:01,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:01,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:58:01,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:01,422 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:58:01,422 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:58:01,422 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:58:01,423 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:58:01,423 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:58:01,423 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:58:01,423 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:58:01,423 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:58:01,423 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:01,423 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:58:01,423 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:58:01,423 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:01,423 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:58:01,423 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:58:01,423 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,423 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:58:01,423 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:58:01,424 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,424 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:01,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:01,425 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,425 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:58:01,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:01,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:01,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:01,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:01,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:58:01,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60893c5e[Count = 0] remaining members to acquire global barrier 2023-04-08 20:58:01,426 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:58:01,426 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,426 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,426 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:58:01,426 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:58:01,426 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,426 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:58:01,426 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:58:01,427 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,427 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:58:01,427 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:58:01,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:01,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:01,427 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:58:01,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:01,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:01,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:01,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:01,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:01,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:58:01,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:58:01,429 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:58:01,429 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:58:01,429 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:58:01,429 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:58:01,429 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:58:01,430 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,430 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:01,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:01,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,430 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:58:01,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:58:01,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:58:01,431 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:01,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:01,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:01,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:01,432 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:01,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:01,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,433 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:58:01,433 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:58:01,433 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:58:01,433 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:58:01,433 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:58:01,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:58:01,433 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:58:01,433 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:01,434 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:58:01,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:58:01,434 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:58:01,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:58:01,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:58:01,434 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:01,434 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,434 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,434 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,434 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,534 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:58:01,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:58:01,536 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:58:01,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:58:01,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:58:01,536 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:58:01,537 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:58:01,537 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:58:01,537 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:58:01,537 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,537 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:01,538 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:58:01,538 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:58:01,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:58:01,538 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,538 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:58:01,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:01,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:58:01,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:01,538 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:58:01,539 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:58:01,539 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:58:01,539 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:58:01,539 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:58:01,539 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:58:01,539 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:58:01,539 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:58:01,539 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:01,539 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:58:01,539 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:58:01,539 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:01,539 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:58:01,539 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:58:01,539 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,539 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:58:01,539 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:58:01,540 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,540 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:01,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:01,540 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,540 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:58:01,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:01,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:01,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:01,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:01,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:58:01,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@581d1420[Count = 0] remaining members to acquire global barrier 2023-04-08 20:58:01,542 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:58:01,542 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,542 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,542 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:58:01,542 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:58:01,542 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:58:01,545 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,545 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:58:01,545 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:58:01,545 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,545 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:58:01,545 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:58:01,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:01,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:01,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:01,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:01,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:01,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:01,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:01,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:58:01,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:58:01,548 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:58:01,548 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:58:01,548 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:58:01,548 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:58:01,548 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:58:01,549 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,549 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:01,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:01,549 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:58:01,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:01,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:58:01,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:58:01,549 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:01,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:01,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:01,550 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:01,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:01,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,552 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,552 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:58:01,552 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:01,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:58:01,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:58:01,552 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:01,552 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,552 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,552 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:58:01,552 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:58:01,552 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,552 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:58:01,552 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:58:01,552 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:58:01,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:58:01,553 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:58:01,553 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:58:01,653 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:58:01,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:58:01,655 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:58:01,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:58:01,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:58:01,656 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:58:01,656 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:58:01,656 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:58:01,656 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:58:01,656 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,656 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:01,657 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,657 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:58:01,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:58:01,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:58:01,657 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,657 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:58:01,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:01,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:58:01,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:01,658 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:58:01,658 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:58:01,658 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:58:01,658 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:58:01,658 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:58:01,658 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:58:01,658 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:58:01,658 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:58:01,658 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:01,658 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:58:01,658 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:58:01,659 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:01,659 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:58:01,659 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:58:01,659 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,659 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:58:01,659 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:58:01,659 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:01,660 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:01,660 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,660 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:58:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:01,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:58:01,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28fc600d[Count = 0] remaining members to acquire global barrier 2023-04-08 20:58:01,661 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:58:01,661 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,661 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,661 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:58:01,661 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,661 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:58:01,661 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:58:01,661 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:58:01,662 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,662 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:58:01,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:01,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:01,662 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:58:01,662 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:58:01,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:01,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:01,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:01,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:01,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:01,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:58:01,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:58:01,664 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:58:01,664 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:58:01,664 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:58:01,664 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:58:01,664 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:58:01,665 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,665 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:01,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:01,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,665 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:58:01,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:01,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:58:01,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:58:01,665 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:01,666 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:01,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:01,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,668 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,668 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:01,668 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:01,668 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:58:01,668 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:58:01,668 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:58:01,668 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:58:01,668 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:58:01,668 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:58:01,668 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:58:01,668 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:58:01,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:58:01,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:58:01,668 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,669 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:01,669 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:58:01,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:58:01,690 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=258 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/C/d3f6cdb27ed1493dac32839b6af7d806 2023-04-08 20:58:01,693 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/A/d17e0b2f780a4d34bc507a2534e6145e as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/d17e0b2f780a4d34bc507a2534e6145e 2023-04-08 20:58:01,696 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/d17e0b2f780a4d34bc507a2534e6145e, entries=150, sequenceid=258, filesize=25.3 K 2023-04-08 20:58:01,697 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/B/6c3201972b0148b1b3aa094050d91198 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/6c3201972b0148b1b3aa094050d91198 2023-04-08 20:58:01,700 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/6c3201972b0148b1b3aa094050d91198, entries=150, sequenceid=258, filesize=11.8 K 2023-04-08 20:58:01,700 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/C/d3f6cdb27ed1493dac32839b6af7d806 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/d3f6cdb27ed1493dac32839b6af7d806 2023-04-08 20:58:01,703 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/d3f6cdb27ed1493dac32839b6af7d806, entries=150, sequenceid=258, filesize=11.8 K 2023-04-08 20:58:01,703 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for e986ad248e3bbfc4197e05257d0f5eac in 1666ms, sequenceid=258, compaction requested=true 2023-04-08 20:58:01,703 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e986ad248e3bbfc4197e05257d0f5eac: 2023-04-08 20:58:01,703 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:58:01,703 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 20:58:01,704 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 20:58:01,703 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:58:01,704 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:58:01,704 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 103942 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 20:58:01,704 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48502 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 20:58:01,704 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1912): e986ad248e3bbfc4197e05257d0f5eac/A is initiating minor compaction (all files) 2023-04-08 20:58:01,705 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): e986ad248e3bbfc4197e05257d0f5eac/B is initiating minor compaction (all files) 2023-04-08 20:58:01,705 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e986ad248e3bbfc4197e05257d0f5eac/A in TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:01,705 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e986ad248e3bbfc4197e05257d0f5eac/B in TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:01,705 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/877b1aa980544e52908cd707237cbc75, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/5652a04eba2b4cc78f4b221e346bd438, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/628561125e744bfaa7903cb06443a8d2, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/d17e0b2f780a4d34bc507a2534e6145e] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp, totalSize=101.5 K 2023-04-08 20:58:01,705 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/dd47585719bf491f8f06d499920c549b, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/543ced48dc984fc7a3e7ee896a777710, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/b37d219dc04140ea87829664ce4a8a8c, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/6c3201972b0148b1b3aa094050d91198] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp, totalSize=47.4 K 2023-04-08 20:58:01,705 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 877b1aa980544e52908cd707237cbc75, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1680987474916 2023-04-08 20:58:01,705 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting dd47585719bf491f8f06d499920c549b, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1680987474916 2023-04-08 20:58:01,705 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 543ced48dc984fc7a3e7ee896a777710, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1680987476093 2023-04-08 20:58:01,705 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 5652a04eba2b4cc78f4b221e346bd438, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1680987476093 2023-04-08 20:58:01,705 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting b37d219dc04140ea87829664ce4a8a8c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1680987478263 2023-04-08 20:58:01,706 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting 628561125e744bfaa7903cb06443a8d2, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1680987478263 2023-04-08 20:58:01,706 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 6c3201972b0148b1b3aa094050d91198, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1680987479396 2023-04-08 20:58:01,706 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] compactions.Compactor(207): Compacting d17e0b2f780a4d34bc507a2534e6145e, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1680987479396 2023-04-08 20:58:01,731 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): e986ad248e3bbfc4197e05257d0f5eac#B#compaction#484 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:58:01,737 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] throttle.PressureAwareThroughputController(145): e986ad248e3bbfc4197e05257d0f5eac#A#compaction#485 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:58:01,769 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:58:01,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:58:01,770 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:58:01,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:58:01,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:58:01,772 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:58:01,772 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:58:01,772 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:58:01,772 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:58:01,772 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,772 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:01,773 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:58:01,773 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:58:01,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:58:01,773 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:01,773 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:58:01,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:01,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:01,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:58:01,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:01,774 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:58:01,774 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:58:01,774 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:58:01,774 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:58:01,774 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:58:01,774 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:58:01,774 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:58:01,774 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:01,774 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:58:01,774 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e986ad248e3bbfc4197e05257d0f5eac 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-08 20:58:01,774 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=A 2023-04-08 20:58:01,775 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:58:01,775 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=B 2023-04-08 20:58:01,775 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:58:01,775 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=C 2023-04-08 20:58:01,775 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:58:02,148 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/B/7fd07d604ca24e58902167d422cae3e7 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/7fd07d604ca24e58902167d422cae3e7 2023-04-08 20:58:02,151 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e986ad248e3bbfc4197e05257d0f5eac/B of e986ad248e3bbfc4197e05257d0f5eac into 7fd07d604ca24e58902167d422cae3e7(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:58:02,151 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e986ad248e3bbfc4197e05257d0f5eac: 2023-04-08 20:58:02,151 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac., storeName=e986ad248e3bbfc4197e05257d0f5eac/B, priority=12, startTime=1680987481703; duration=0sec 2023-04-08 20:58:02,151 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 20:58:02,151 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 20:58:02,152 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48502 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 20:58:02,152 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1912): e986ad248e3bbfc4197e05257d0f5eac/C is initiating minor compaction (all files) 2023-04-08 20:58:02,152 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e986ad248e3bbfc4197e05257d0f5eac/C in TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:02,152 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/e260fed857474dc8b19f37a7df80b5b3, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/668ee8fba45a4a37a1161a03dd3b2204, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/c38d5e7d7c1f441c8141f6ce475fcefd, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/d3f6cdb27ed1493dac32839b6af7d806] into tmpdir=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp, totalSize=47.4 K 2023-04-08 20:58:02,152 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting e260fed857474dc8b19f37a7df80b5b3, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1680987474916 2023-04-08 20:58:02,153 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting 668ee8fba45a4a37a1161a03dd3b2204, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1680987476093 2023-04-08 20:58:02,153 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting c38d5e7d7c1f441c8141f6ce475fcefd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1680987478263 2023-04-08 20:58:02,153 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] compactions.Compactor(207): Compacting d3f6cdb27ed1493dac32839b6af7d806, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1680987479396 2023-04-08 20:58:02,158 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] throttle.PressureAwareThroughputController(145): e986ad248e3bbfc4197e05257d0f5eac#C#compaction#487 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 20:58:02,168 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/C/513e25b361fe4411bc173fc3b2463819 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/513e25b361fe4411bc173fc3b2463819 2023-04-08 20:58:02,171 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e986ad248e3bbfc4197e05257d0f5eac/C of e986ad248e3bbfc4197e05257d0f5eac into 513e25b361fe4411bc173fc3b2463819(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 20:58:02,171 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e986ad248e3bbfc4197e05257d0f5eac: 2023-04-08 20:58:02,171 INFO [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac., storeName=e986ad248e3bbfc4197e05257d0f5eac/C, priority=12, startTime=1680987481704; duration=0sec 2023-04-08 20:58:02,171 DEBUG [RS:0;jenkins-hbase20:41091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:58:02,185 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040841c25bc7f1f74ca69271092946ed5455 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040841c25bc7f1f74ca69271092946ed5455 2023-04-08 20:58:02,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:58:02,194 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:58:02,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680987482230, max timestamp=1680987482236Num uniques 859; 2023-04-08 20:58:02,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 20:58:02,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680987482230, max timestamp=1680987482236Num uniques 859; 2023-04-08 20:58:02,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 20:58:02,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680987482230, max timestamp=1680987482236Num uniques 859; 2023-04-08 20:58:02,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 20:58:02,283 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:58:02,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39370 deadline: 1680987542280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:02,387 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:58:02,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39370 deadline: 1680987542384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:02,592 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:58:02,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39370 deadline: 1680987542588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:02,608 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=269, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/A/78a7ade3182345eba311f0c98064ee5f 2023-04-08 20:58:02,894 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:58:02,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41091] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39370 deadline: 1680987542893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:02,971 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/A/4b0173112c4f4d7584d4ce03c27dae13 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/4b0173112c4f4d7584d4ce03c27dae13 2023-04-08 20:58:02,974 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e986ad248e3bbfc4197e05257d0f5eac/A of e986ad248e3bbfc4197e05257d0f5eac into 4b0173112c4f4d7584d4ce03c27dae13(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-08 20:58:02,974 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e986ad248e3bbfc4197e05257d0f5eac: 2023-04-08 20:58:02,974 INFO [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac., storeName=e986ad248e3bbfc4197e05257d0f5eac/A, priority=12, startTime=1680987481703; duration=1sec 2023-04-08 20:58:02,975 DEBUG [RS:0;jenkins-hbase20:41091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 20:58:03,016 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/B/671672735a124b5dba040b9025eb48a4 2023-04-08 20:58:03,399 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 20:58:03,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39370 deadline: 1680987543398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:03,432 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/C/19068ebe288b49fe853ecd916b0b6812 2023-04-08 20:58:03,435 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/A/78a7ade3182345eba311f0c98064ee5f as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/78a7ade3182345eba311f0c98064ee5f 2023-04-08 20:58:03,438 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/78a7ade3182345eba311f0c98064ee5f, entries=150, sequenceid=269, filesize=25.4 K 2023-04-08 20:58:03,438 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/B/671672735a124b5dba040b9025eb48a4 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/671672735a124b5dba040b9025eb48a4 2023-04-08 20:58:03,441 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/671672735a124b5dba040b9025eb48a4, entries=150, sequenceid=269, filesize=11.9 K 2023-04-08 20:58:03,442 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/C/19068ebe288b49fe853ecd916b0b6812 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/19068ebe288b49fe853ecd916b0b6812 2023-04-08 20:58:03,444 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/19068ebe288b49fe853ecd916b0b6812, entries=150, sequenceid=269, filesize=11.9 K 2023-04-08 20:58:03,445 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for e986ad248e3bbfc4197e05257d0f5eac in 1671ms, sequenceid=269, compaction requested=false 2023-04-08 20:58:03,445 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e986ad248e3bbfc4197e05257d0f5eac: 2023-04-08 20:58:03,445 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:03,445 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:58:03,445 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:58:03,445 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:03,446 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:58:03,446 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:58:03,446 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:03,446 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:03,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:03,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:03,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:03,447 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:03,447 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:58:03,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:03,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:03,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:03,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:03,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:03,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:58:03,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30356f08[Count = 0] remaining members to acquire global barrier 2023-04-08 20:58:03,448 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:58:03,448 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:03,449 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:03,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:03,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:03,449 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:58:03,449 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:58:03,449 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:58:03,449 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:03,449 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:58:03,450 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:03,450 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:58:03,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:03,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:03,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:03,450 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:58:03,450 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:58:03,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:03,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:03,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:03,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:03,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:03,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:03,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:03,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:58:03,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:58:03,452 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:58:03,452 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:58:03,452 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:58:03,452 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:58:03,452 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:58:03,453 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:03,453 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:03,453 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:58:03,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:03,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:03,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:03,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:03,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:03,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:58:03,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:58:03,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:03,454 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:03,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:03,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:03,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:03,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:03,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:03,455 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:03,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:03,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:03,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:03,457 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:03,457 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:03,457 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:58:03,457 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:58:03,457 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:58:03,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:58:03,457 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:58:03,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:58:03,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:58:03,457 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:58:03,457 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:03,457 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:58:03,457 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:03,457 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:58:03,457 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:03,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:58:03,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:58:03,458 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:03,458 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:03,558 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:58:03,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:58:03,559 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-08 20:58:03,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 20:58:03,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 20:58:03,560 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 20:58:03,560 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:58:03,560 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 20:58:03,560 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 20:58:03,560 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:03,561 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:03,561 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:58:03,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:58:03,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:58:03,561 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:03,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:03,561 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:03,561 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 20:58:03,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:03,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 20:58:03,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:03,562 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 20:58:03,562 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 20:58:03,562 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 20:58:03,562 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 20:58:03,562 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 20:58:03,562 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 20:58:03,562 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 20:58:03,562 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 20:58:03,562 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:03,562 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. started... 2023-04-08 20:58:03,562 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e986ad248e3bbfc4197e05257d0f5eac 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-08 20:58:03,563 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=A 2023-04-08 20:58:03,563 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:58:03,563 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=B 2023-04-08 20:58:03,563 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:58:03,563 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=C 2023-04-08 20:58:03,563 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 20:58:03,946 DEBUG [Thread-3377] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x44a601bf to 127.0.0.1:52440 2023-04-08 20:58:03,946 DEBUG [Thread-3377] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:58:03,947 DEBUG [Thread-3379] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x67bf0d7e to 127.0.0.1:52440 2023-04-08 20:58:03,947 DEBUG [Thread-3383] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x509de0a9 to 127.0.0.1:52440 2023-04-08 20:58:03,947 DEBUG [Thread-3379] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:58:03,947 DEBUG [Thread-3383] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:58:03,948 DEBUG [Thread-3385] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x17e16418 to 127.0.0.1:52440 2023-04-08 20:58:03,948 DEBUG [Thread-3385] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:58:03,948 DEBUG [Thread-3381] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3d52316c to 127.0.0.1:52440 2023-04-08 20:58:03,948 DEBUG [Thread-3381] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:58:03,973 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408e6c0b5264f2f4d8ba89a8000da4b9048 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408e6c0b5264f2f4d8ba89a8000da4b9048 2023-04-08 20:58:04,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=41091] regionserver.HRegion(9158): Flush requested on e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:58:04,339 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. as already flushing 2023-04-08 20:58:04,339 DEBUG [Thread-3374] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x407f33a2 to 127.0.0.1:52440 2023-04-08 20:58:04,339 DEBUG [Thread-3374] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:58:04,379 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=308, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/A/3c99ea4c4f9245c7aedd3d5b064379eb 2023-04-08 20:58:04,405 DEBUG [Thread-3372] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x112588a6 to 127.0.0.1:52440 2023-04-08 20:58:04,405 DEBUG [Thread-3372] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:58:04,799 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=308 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/B/fa711877710e444bb72e5d1198b12615 2023-04-08 20:58:05,215 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=308 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/C/72717d893b414790ad5d646dbb2d8282 2023-04-08 20:58:05,224 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/A/3c99ea4c4f9245c7aedd3d5b064379eb as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/3c99ea4c4f9245c7aedd3d5b064379eb 2023-04-08 20:58:05,229 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/3c99ea4c4f9245c7aedd3d5b064379eb, entries=150, sequenceid=308, filesize=25.4 K 2023-04-08 20:58:05,230 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/B/fa711877710e444bb72e5d1198b12615 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/fa711877710e444bb72e5d1198b12615 2023-04-08 20:58:05,234 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/fa711877710e444bb72e5d1198b12615, entries=150, sequenceid=308, filesize=11.9 K 2023-04-08 20:58:05,235 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/C/72717d893b414790ad5d646dbb2d8282 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/72717d893b414790ad5d646dbb2d8282 2023-04-08 20:58:05,239 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/72717d893b414790ad5d646dbb2d8282, entries=150, sequenceid=308, filesize=11.9 K 2023-04-08 20:58:05,240 INFO [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=13.42 KB/13740 for e986ad248e3bbfc4197e05257d0f5eac in 1677ms, sequenceid=308, compaction requested=true 2023-04-08 20:58:05,240 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e986ad248e3bbfc4197e05257d0f5eac: 2023-04-08 20:58:05,240 DEBUG [rs(jenkins-hbase20.apache.org,41091,1680987313601)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:05,240 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 20:58:05,240 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 20:58:05,240 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:05,240 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 20:58:05,240 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 20:58:05,241 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:05,241 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:05,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:05,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:05,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:05,241 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:05,242 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 20:58:05,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:05,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:05,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:05,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:05,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:05,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,41091,1680987313601' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 20:58:05,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1977de9c[Count = 0] remaining members to acquire global barrier 2023-04-08 20:58:05,244 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 20:58:05,244 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:05,245 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:05,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:05,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:05,245 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 20:58:05,245 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 20:58:05,245 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,41091,1680987313601' in zk 2023-04-08 20:58:05,245 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:05,245 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 20:58:05,246 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:05,246 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 20:58:05,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:05,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:05,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:05,247 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:58:05,247 DEBUG [member: 'jenkins-hbase20.apache.org,41091,1680987313601' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 20:58:05,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:05,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:05,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:05,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:05,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:05,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:05,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:05,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,41091,1680987313601': 2023-04-08 20:58:05,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,41091,1680987313601' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 20:58:05,249 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 20:58:05,250 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 20:58:05,250 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 20:58:05,250 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 20:58:05,250 INFO [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 20:58:05,251 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:05,251 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:05,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:05,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 20:58:05,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 20:58:05,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:05,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:05,251 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:58:05,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 20:58:05,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:58:05,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:58:05,252 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:05,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:05,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:05,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 20:58:05,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:05,253 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:05,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:05,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 20:58:05,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 20:58:05,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:05,255 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:05,255 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 20:58:05,255 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:05,255 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 20:58:05,255 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 20:58:05,255 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 20:58:05,255 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 20:58:05,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 20:58:05,256 DEBUG [Thread-3376] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 20:58:05,256 DEBUG [Thread-3376] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 20:58:05,255 DEBUG [(jenkins-hbase20.apache.org,46121,1680987312586)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 20:58:05,255 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 20:58:05,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 20:58:05,255 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:05,256 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:05,256 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 20:58:05,256 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 20:58:05,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 20:58:05,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 20:58:05,356 DEBUG [Thread-3376] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 20:58:05,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 20:58:06,462 DEBUG [Thread-3366] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x323c0ce8 to 127.0.0.1:52440 2023-04-08 20:58:06,462 DEBUG [Thread-3366] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:58:06,499 DEBUG [Thread-3368] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0754289a to 127.0.0.1:52440 2023-04-08 20:58:06,499 DEBUG [Thread-3368] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:58:10,368 DEBUG [Thread-3370] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3d853d1b to 127.0.0.1:52440 2023-04-08 20:58:10,369 DEBUG [Thread-3370] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:58:10,369 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-08 20:58:10,369 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(394): wrote 21 2023-04-08 20:58:10,370 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(394): wrote 19 2023-04-08 20:58:10,370 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(394): wrote 35 2023-04-08 20:58:10,370 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(394): wrote 172 2023-04-08 20:58:10,370 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(394): wrote 14 2023-04-08 20:58:10,370 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-08 20:58:10,370 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(398): read 5182 2023-04-08 20:58:10,370 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(398): read 5262 2023-04-08 20:58:10,371 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(398): read 5254 2023-04-08 20:58:10,371 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(398): read 5203 2023-04-08 20:58:10,371 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(398): read 5271 2023-04-08 20:58:10,371 INFO [Listener at localhost.localdomain/41823] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-08 20:58:10,371 INFO [Listener at localhost.localdomain/41823] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-08 20:58:10,372 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5054e06c to 127.0.0.1:52440 2023-04-08 20:58:10,372 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:58:10,373 INFO [Listener at localhost.localdomain/41823] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-08 20:58:10,374 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-08 20:58:10,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-08 20:58:10,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-08 20:58:10,379 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680987490379"}]},"ts":"1680987490379"} 2023-04-08 20:58:10,380 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-08 20:58:10,396 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-08 20:58:10,397 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e986ad248e3bbfc4197e05257d0f5eac, UNASSIGN}] 2023-04-08 20:58:10,400 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e986ad248e3bbfc4197e05257d0f5eac, UNASSIGN 2023-04-08 20:58:10,401 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=e986ad248e3bbfc4197e05257d0f5eac, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:10,401 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987490401"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680987490401"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680987490401"}]},"ts":"1680987490401"} 2023-04-08 20:58:10,403 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601}] 2023-04-08 20:58:10,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-08 20:58:10,556 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:58:10,556 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing e986ad248e3bbfc4197e05257d0f5eac, disabling compactions & flushes 2023-04-08 20:58:10,556 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:10,556 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:10,556 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. after waiting 0 ms 2023-04-08 20:58:10,556 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:10,556 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing e986ad248e3bbfc4197e05257d0f5eac 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-08 20:58:10,556 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=A 2023-04-08 20:58:10,556 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:58:10,556 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=B 2023-04-08 20:58:10,556 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:58:10,556 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e986ad248e3bbfc4197e05257d0f5eac, store=C 2023-04-08 20:58:10,556 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 20:58:10,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-08 20:58:10,969 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304082bd3a03050e844e7b99e63c9fb1ad0be to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304082bd3a03050e844e7b99e63c9fb1ad0be 2023-04-08 20:58:10,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-08 20:58:11,379 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=316, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/A/809105c6ec96463782b5c3c9159c4737 2023-04-08 20:58:11,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-08 20:58:11,800 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=316 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/B/0b16b04a4fe74a37a195ac2848025b89 2023-04-08 20:58:12,216 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=316 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/C/f551e65ac3c44892bf955e2bee03f926 2023-04-08 20:58:12,226 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/A/809105c6ec96463782b5c3c9159c4737 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/809105c6ec96463782b5c3c9159c4737 2023-04-08 20:58:12,233 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/809105c6ec96463782b5c3c9159c4737, entries=150, sequenceid=316, filesize=25.4 K 2023-04-08 20:58:12,234 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/B/0b16b04a4fe74a37a195ac2848025b89 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/0b16b04a4fe74a37a195ac2848025b89 2023-04-08 20:58:12,240 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/0b16b04a4fe74a37a195ac2848025b89, entries=150, sequenceid=316, filesize=11.9 K 2023-04-08 20:58:12,241 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/.tmp/C/f551e65ac3c44892bf955e2bee03f926 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/f551e65ac3c44892bf955e2bee03f926 2023-04-08 20:58:12,247 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/f551e65ac3c44892bf955e2bee03f926, entries=150, sequenceid=316, filesize=11.9 K 2023-04-08 20:58:12,247 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for e986ad248e3bbfc4197e05257d0f5eac in 1691ms, sequenceid=316, compaction requested=true 2023-04-08 20:58:12,263 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/8c7834a3e2b0497fb62833f485237746, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/39fbfc5b9f1d45db951c704a49236afe, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/b7a7fae04c134dcab05eeacc02a6266a, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/1df5438db2c84d41a1bde13f7f90d8b6, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/b41833a47369456cb3bacb40b070d410, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/89ef2fd6787b4922a37291f8539ca12c, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/268070a5d01144699275527c6e320e08, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/5a3505bca21f44198e22df7de45bcd27, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/a24a0ca46d1d44a8a21f2d7f860e989c, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/877b1aa980544e52908cd707237cbc75, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/5652a04eba2b4cc78f4b221e346bd438, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/628561125e744bfaa7903cb06443a8d2, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/d17e0b2f780a4d34bc507a2534e6145e] to archive 2023-04-08 20:58:12,264 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 20:58:12,265 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/8c7834a3e2b0497fb62833f485237746 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/8c7834a3e2b0497fb62833f485237746 2023-04-08 20:58:12,266 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/39fbfc5b9f1d45db951c704a49236afe to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/39fbfc5b9f1d45db951c704a49236afe 2023-04-08 20:58:12,267 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/b7a7fae04c134dcab05eeacc02a6266a to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/b7a7fae04c134dcab05eeacc02a6266a 2023-04-08 20:58:12,268 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/1df5438db2c84d41a1bde13f7f90d8b6 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/1df5438db2c84d41a1bde13f7f90d8b6 2023-04-08 20:58:12,268 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/b41833a47369456cb3bacb40b070d410 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/b41833a47369456cb3bacb40b070d410 2023-04-08 20:58:12,269 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/89ef2fd6787b4922a37291f8539ca12c to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/89ef2fd6787b4922a37291f8539ca12c 2023-04-08 20:58:12,270 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/268070a5d01144699275527c6e320e08 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/268070a5d01144699275527c6e320e08 2023-04-08 20:58:12,271 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/5a3505bca21f44198e22df7de45bcd27 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/5a3505bca21f44198e22df7de45bcd27 2023-04-08 20:58:12,272 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/a24a0ca46d1d44a8a21f2d7f860e989c to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/a24a0ca46d1d44a8a21f2d7f860e989c 2023-04-08 20:58:12,273 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/877b1aa980544e52908cd707237cbc75 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/877b1aa980544e52908cd707237cbc75 2023-04-08 20:58:12,274 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/5652a04eba2b4cc78f4b221e346bd438 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/5652a04eba2b4cc78f4b221e346bd438 2023-04-08 20:58:12,275 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/628561125e744bfaa7903cb06443a8d2 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/628561125e744bfaa7903cb06443a8d2 2023-04-08 20:58:12,276 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/d17e0b2f780a4d34bc507a2534e6145e to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/d17e0b2f780a4d34bc507a2534e6145e 2023-04-08 20:58:12,290 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/72dc29331ded40d08416a8a884400312, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/cd573042af1842a7a9c4a5fb3e5bc6fc, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/0de7c8d866c041209911a04fe1c18192, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/7fab73ecc11f44e8a2fba15761dede3c, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/7c91e95173484b28886611bdee7822fb, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/fdde235952c64d7193f5e601145b538b, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/d026f50e05c94cd2bb8800df671177c6, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/77d293431ac74e4aaf86ccf4cd14c249, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/dd47585719bf491f8f06d499920c549b, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/62e0fa63ff704089a95502369af257bc, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/543ced48dc984fc7a3e7ee896a777710, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/b37d219dc04140ea87829664ce4a8a8c, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/6c3201972b0148b1b3aa094050d91198] to archive 2023-04-08 20:58:12,291 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 20:58:12,293 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/72dc29331ded40d08416a8a884400312 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/72dc29331ded40d08416a8a884400312 2023-04-08 20:58:12,294 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/cd573042af1842a7a9c4a5fb3e5bc6fc to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/cd573042af1842a7a9c4a5fb3e5bc6fc 2023-04-08 20:58:12,294 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/0de7c8d866c041209911a04fe1c18192 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/0de7c8d866c041209911a04fe1c18192 2023-04-08 20:58:12,295 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/7fab73ecc11f44e8a2fba15761dede3c to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/7fab73ecc11f44e8a2fba15761dede3c 2023-04-08 20:58:12,296 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/7c91e95173484b28886611bdee7822fb to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/7c91e95173484b28886611bdee7822fb 2023-04-08 20:58:12,296 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/fdde235952c64d7193f5e601145b538b to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/fdde235952c64d7193f5e601145b538b 2023-04-08 20:58:12,297 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/d026f50e05c94cd2bb8800df671177c6 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/d026f50e05c94cd2bb8800df671177c6 2023-04-08 20:58:12,298 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/77d293431ac74e4aaf86ccf4cd14c249 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/77d293431ac74e4aaf86ccf4cd14c249 2023-04-08 20:58:12,299 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/dd47585719bf491f8f06d499920c549b to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/dd47585719bf491f8f06d499920c549b 2023-04-08 20:58:12,300 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/62e0fa63ff704089a95502369af257bc to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/62e0fa63ff704089a95502369af257bc 2023-04-08 20:58:12,300 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/543ced48dc984fc7a3e7ee896a777710 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/543ced48dc984fc7a3e7ee896a777710 2023-04-08 20:58:12,301 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/b37d219dc04140ea87829664ce4a8a8c to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/b37d219dc04140ea87829664ce4a8a8c 2023-04-08 20:58:12,302 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/6c3201972b0148b1b3aa094050d91198 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/6c3201972b0148b1b3aa094050d91198 2023-04-08 20:58:12,303 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/e65543d837f940a49870fed5e1ed360b, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/1837a5958ab34dcd8b72ab59fcc44e27, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/d7cca22e678a4c828c1d204a48518955, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/ecc7240eb5264597bbe3a85160fab2cf, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/7329a7d4e110480e85f639e9714bcb88, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/fafdf2a87caf4e2588ef70fcff4855ee, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/3da986f783534e5792a4bc77536a5c77, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/86187b52c12441e9a5cce74e48084aac, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/e260fed857474dc8b19f37a7df80b5b3, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/142ef174e9d34176809140bd7753fed1, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/668ee8fba45a4a37a1161a03dd3b2204, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/c38d5e7d7c1f441c8141f6ce475fcefd, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/d3f6cdb27ed1493dac32839b6af7d806] to archive 2023-04-08 20:58:12,304 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 20:58:12,305 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/e65543d837f940a49870fed5e1ed360b to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/e65543d837f940a49870fed5e1ed360b 2023-04-08 20:58:12,306 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/1837a5958ab34dcd8b72ab59fcc44e27 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/1837a5958ab34dcd8b72ab59fcc44e27 2023-04-08 20:58:12,306 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/d7cca22e678a4c828c1d204a48518955 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/d7cca22e678a4c828c1d204a48518955 2023-04-08 20:58:12,307 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/ecc7240eb5264597bbe3a85160fab2cf to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/ecc7240eb5264597bbe3a85160fab2cf 2023-04-08 20:58:12,308 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/7329a7d4e110480e85f639e9714bcb88 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/7329a7d4e110480e85f639e9714bcb88 2023-04-08 20:58:12,309 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/fafdf2a87caf4e2588ef70fcff4855ee to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/fafdf2a87caf4e2588ef70fcff4855ee 2023-04-08 20:58:12,309 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/3da986f783534e5792a4bc77536a5c77 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/3da986f783534e5792a4bc77536a5c77 2023-04-08 20:58:12,310 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/86187b52c12441e9a5cce74e48084aac to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/86187b52c12441e9a5cce74e48084aac 2023-04-08 20:58:12,311 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/e260fed857474dc8b19f37a7df80b5b3 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/e260fed857474dc8b19f37a7df80b5b3 2023-04-08 20:58:12,311 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/142ef174e9d34176809140bd7753fed1 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/142ef174e9d34176809140bd7753fed1 2023-04-08 20:58:12,312 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/668ee8fba45a4a37a1161a03dd3b2204 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/668ee8fba45a4a37a1161a03dd3b2204 2023-04-08 20:58:12,313 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/c38d5e7d7c1f441c8141f6ce475fcefd to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/c38d5e7d7c1f441c8141f6ce475fcefd 2023-04-08 20:58:12,314 DEBUG [StoreCloser-TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/d3f6cdb27ed1493dac32839b6af7d806 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/d3f6cdb27ed1493dac32839b6af7d806 2023-04-08 20:58:12,317 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/recovered.edits/319.seqid, newMaxSeqId=319, maxSeqId=4 2023-04-08 20:58:12,318 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac. 2023-04-08 20:58:12,318 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for e986ad248e3bbfc4197e05257d0f5eac: 2023-04-08 20:58:12,319 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:58:12,320 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=e986ad248e3bbfc4197e05257d0f5eac, regionState=CLOSED 2023-04-08 20:58:12,320 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680987492320"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680987492320"}]},"ts":"1680987492320"} 2023-04-08 20:58:12,322 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-08 20:58:12,322 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure e986ad248e3bbfc4197e05257d0f5eac, server=jenkins-hbase20.apache.org,41091,1680987313601 in 1.9180 sec 2023-04-08 20:58:12,323 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-08 20:58:12,323 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e986ad248e3bbfc4197e05257d0f5eac, UNASSIGN in 1.9250 sec 2023-04-08 20:58:12,323 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680987492323"}]},"ts":"1680987492323"} 2023-04-08 20:58:12,324 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-08 20:58:12,325 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-08 20:58:12,326 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9510 sec 2023-04-08 20:58:12,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-08 20:58:12,488 INFO [Listener at localhost.localdomain/41823] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-08 20:58:12,489 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-08 20:58:12,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 20:58:12,493 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 20:58:12,495 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 20:58:12,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-08 20:58:12,501 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:58:12,504 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A, FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B, FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C, FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/recovered.edits] 2023-04-08 20:58:12,507 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/3c99ea4c4f9245c7aedd3d5b064379eb to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/3c99ea4c4f9245c7aedd3d5b064379eb 2023-04-08 20:58:12,508 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/4b0173112c4f4d7584d4ce03c27dae13 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/4b0173112c4f4d7584d4ce03c27dae13 2023-04-08 20:58:12,509 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/78a7ade3182345eba311f0c98064ee5f to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/78a7ade3182345eba311f0c98064ee5f 2023-04-08 20:58:12,511 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/809105c6ec96463782b5c3c9159c4737 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/A/809105c6ec96463782b5c3c9159c4737 2023-04-08 20:58:12,513 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/0b16b04a4fe74a37a195ac2848025b89 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/0b16b04a4fe74a37a195ac2848025b89 2023-04-08 20:58:12,515 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/671672735a124b5dba040b9025eb48a4 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/671672735a124b5dba040b9025eb48a4 2023-04-08 20:58:12,517 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/7fd07d604ca24e58902167d422cae3e7 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/7fd07d604ca24e58902167d422cae3e7 2023-04-08 20:58:12,518 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/fa711877710e444bb72e5d1198b12615 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/B/fa711877710e444bb72e5d1198b12615 2023-04-08 20:58:12,521 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/19068ebe288b49fe853ecd916b0b6812 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/19068ebe288b49fe853ecd916b0b6812 2023-04-08 20:58:12,523 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/513e25b361fe4411bc173fc3b2463819 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/513e25b361fe4411bc173fc3b2463819 2023-04-08 20:58:12,525 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/72717d893b414790ad5d646dbb2d8282 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/72717d893b414790ad5d646dbb2d8282 2023-04-08 20:58:12,526 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/f551e65ac3c44892bf955e2bee03f926 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/C/f551e65ac3c44892bf955e2bee03f926 2023-04-08 20:58:12,530 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/recovered.edits/319.seqid to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac/recovered.edits/319.seqid 2023-04-08 20:58:12,530 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/.tmp/data/default/TestAcidGuarantees/e986ad248e3bbfc4197e05257d0f5eac 2023-04-08 20:58:12,531 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-08 20:58:12,531 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-08 20:58:12,532 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-08 20:58:12,536 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304080fd4d5f9e8104df39105ee26107c293f to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304080fd4d5f9e8104df39105ee26107c293f 2023-04-08 20:58:12,538 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304082bd3a03050e844e7b99e63c9fb1ad0be to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304082bd3a03050e844e7b99e63c9fb1ad0be 2023-04-08 20:58:12,540 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040841c25bc7f1f74ca69271092946ed5455 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040841c25bc7f1f74ca69271092946ed5455 2023-04-08 20:58:12,541 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040847fdd941d2c946b98c6e7be1a09126d2 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040847fdd941d2c946b98c6e7be1a09126d2 2023-04-08 20:58:12,543 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040855067c13e5f64ef8a0394314065d3aaa to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040855067c13e5f64ef8a0394314065d3aaa 2023-04-08 20:58:12,545 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304085c0ddc9218b44f13a40779cfad361fb8 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304085c0ddc9218b44f13a40779cfad361fb8 2023-04-08 20:58:12,546 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304087c71218294074bada12f5fa41b3e88ae to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304087c71218294074bada12f5fa41b3e88ae 2023-04-08 20:58:12,547 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040880b738c7d0a6499d8f63996a1991c447 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040880b738c7d0a6499d8f63996a1991c447 2023-04-08 20:58:12,548 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040887f0f8c94b3140c581f1d93ec193719b to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040887f0f8c94b3140c581f1d93ec193719b 2023-04-08 20:58:12,548 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040896c416f74dc84b9185f764634becf76d to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040896c416f74dc84b9185f764634becf76d 2023-04-08 20:58:12,549 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408a6436ff7b935411ca5e983095b7373ce to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408a6436ff7b935411ca5e983095b7373ce 2023-04-08 20:58:12,550 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408b9b5fc464d674a7a9519e4e1084f7022 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408b9b5fc464d674a7a9519e4e1084f7022 2023-04-08 20:58:12,551 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408c1465650c5c7444aa2fc78f06cb71aed to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408c1465650c5c7444aa2fc78f06cb71aed 2023-04-08 20:58:12,551 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408e6c0b5264f2f4d8ba89a8000da4b9048 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408e6c0b5264f2f4d8ba89a8000da4b9048 2023-04-08 20:58:12,552 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408fbcd3ffe174f4939a5103c68a8cfe782 to hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408fbcd3ffe174f4939a5103c68a8cfe782 2023-04-08 20:58:12,552 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-08 20:58:12,554 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 20:58:12,555 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-08 20:58:12,556 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-08 20:58:12,557 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 20:58:12,557 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-08 20:58:12,557 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680987492557"}]},"ts":"9223372036854775807"} 2023-04-08 20:58:12,558 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-08 20:58:12,559 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => e986ad248e3bbfc4197e05257d0f5eac, NAME => 'TestAcidGuarantees,,1680987460811.e986ad248e3bbfc4197e05257d0f5eac.', STARTKEY => '', ENDKEY => ''}] 2023-04-08 20:58:12,559 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-08 20:58:12,559 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680987492559"}]},"ts":"9223372036854775807"} 2023-04-08 20:58:12,561 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-08 20:58:12,562 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 20:58:12,563 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 73 msec 2023-04-08 20:58:12,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46121] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-08 20:58:12,597 INFO [Listener at localhost.localdomain/41823] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-08 20:58:12,613 INFO [Listener at localhost.localdomain/41823] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225 (was 225), OpenFileDescriptor=508 (was 544), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=305 (was 303) - SystemLoadAverage LEAK? -, ProcessCount=166 (was 166), AvailableMemoryMB=3648 (was 3591) - AvailableMemoryMB LEAK? - 2023-04-08 20:58:12,614 INFO [Listener at localhost.localdomain/41823] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-08 20:58:12,614 INFO [Listener at localhost.localdomain/41823] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-08 20:58:12,614 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4c9bc423 to 127.0.0.1:52440 2023-04-08 20:58:12,614 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:58:12,615 DEBUG [Listener at localhost.localdomain/41823] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-08 20:58:12,615 DEBUG [Listener at localhost.localdomain/41823] util.JVMClusterUtil(257): Found active master hash=500107338, stopped=false 2023-04-08 20:58:12,615 INFO [Listener at localhost.localdomain/41823] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase20.apache.org,46121,1680987312586 2023-04-08 20:58:12,616 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-08 20:58:12,616 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-08 20:58:12,616 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-08 20:58:12,616 INFO [Listener at localhost.localdomain/41823] procedure2.ProcedureExecutor(629): Stopping 2023-04-08 20:58:12,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-08 20:58:12,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-08 20:58:12,617 DEBUG [Listener at localhost.localdomain/41823] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x75b4962c to 127.0.0.1:52440 2023-04-08 20:58:12,617 DEBUG [Listener at localhost.localdomain/41823] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:58:12,617 INFO [Listener at localhost.localdomain/41823] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase20.apache.org,41091,1680987313601' ***** 2023-04-08 20:58:12,617 INFO [Listener at localhost.localdomain/41823] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-08 20:58:12,617 INFO [RS:0;jenkins-hbase20:41091] regionserver.HeapMemoryManager(220): Stopping 2023-04-08 20:58:12,617 INFO [RS:0;jenkins-hbase20:41091] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-08 20:58:12,617 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-08 20:58:12,617 INFO [RS:0;jenkins-hbase20:41091] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-08 20:58:12,618 INFO [RS:0;jenkins-hbase20:41091] regionserver.HRegionServer(3304): Received CLOSE for 301db7cfa588945a1b29ec427fecfa07 2023-04-08 20:58:12,618 INFO [RS:0;jenkins-hbase20:41091] regionserver.HRegionServer(1145): stopping server jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:12,618 DEBUG [RS:0;jenkins-hbase20:41091] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5109cd44 to 127.0.0.1:52440 2023-04-08 20:58:12,619 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 301db7cfa588945a1b29ec427fecfa07, disabling compactions & flushes 2023-04-08 20:58:12,619 DEBUG [RS:0;jenkins-hbase20:41091] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:58:12,619 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680987315735.301db7cfa588945a1b29ec427fecfa07. 2023-04-08 20:58:12,619 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680987315735.301db7cfa588945a1b29ec427fecfa07. 2023-04-08 20:58:12,619 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680987315735.301db7cfa588945a1b29ec427fecfa07. after waiting 0 ms 2023-04-08 20:58:12,619 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680987315735.301db7cfa588945a1b29ec427fecfa07. 2023-04-08 20:58:12,619 INFO [RS:0;jenkins-hbase20:41091] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-08 20:58:12,619 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 301db7cfa588945a1b29ec427fecfa07 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-08 20:58:12,619 INFO [RS:0;jenkins-hbase20:41091] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-08 20:58:12,619 INFO [RS:0;jenkins-hbase20:41091] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-08 20:58:12,619 INFO [RS:0;jenkins-hbase20:41091] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-08 20:58:12,619 INFO [RS:0;jenkins-hbase20:41091] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-08 20:58:12,620 DEBUG [RS:0;jenkins-hbase20:41091] regionserver.HRegionServer(1479): Online Regions={301db7cfa588945a1b29ec427fecfa07=hbase:namespace,,1680987315735.301db7cfa588945a1b29ec427fecfa07., 1588230740=hbase:meta,,1.1588230740} 2023-04-08 20:58:12,620 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-08 20:58:12,620 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-08 20:58:12,620 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-08 20:58:12,620 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-08 20:58:12,620 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-08 20:58:12,620 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.94 KB heapSize=36.31 KB 2023-04-08 20:58:12,622 DEBUG [RS:0;jenkins-hbase20:41091] regionserver.HRegionServer(1505): Waiting on 1588230740, 301db7cfa588945a1b29ec427fecfa07 2023-04-08 20:58:12,626 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.29 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/meta/1588230740/.tmp/info/cfa55d19d8f345439bfd77bdf44d3547 2023-04-08 20:58:12,631 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for cfa55d19d8f345439bfd77bdf44d3547 2023-04-08 20:58:12,709 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-08 20:58:12,781 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-08 20:58:12,781 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-08 20:58:12,822 DEBUG [RS:0;jenkins-hbase20:41091] regionserver.HRegionServer(1505): Waiting on 1588230740, 301db7cfa588945a1b29ec427fecfa07 2023-04-08 20:58:13,022 DEBUG [RS:0;jenkins-hbase20:41091] regionserver.HRegionServer(1505): Waiting on 1588230740, 301db7cfa588945a1b29ec427fecfa07 2023-04-08 20:58:13,027 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/namespace/301db7cfa588945a1b29ec427fecfa07/.tmp/info/7fde6c097c5c4391a65c955e8f1169ce 2023-04-08 20:58:13,037 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/meta/1588230740/.tmp/rep_barrier/622c86e7957445aa8cabd9e01729e063 2023-04-08 20:58:13,039 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/namespace/301db7cfa588945a1b29ec427fecfa07/.tmp/info/7fde6c097c5c4391a65c955e8f1169ce as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/namespace/301db7cfa588945a1b29ec427fecfa07/info/7fde6c097c5c4391a65c955e8f1169ce 2023-04-08 20:58:13,044 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 622c86e7957445aa8cabd9e01729e063 2023-04-08 20:58:13,045 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/namespace/301db7cfa588945a1b29ec427fecfa07/info/7fde6c097c5c4391a65c955e8f1169ce, entries=2, sequenceid=6, filesize=4.8 K 2023-04-08 20:58:13,046 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 301db7cfa588945a1b29ec427fecfa07 in 427ms, sequenceid=6, compaction requested=false 2023-04-08 20:58:13,052 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/namespace/301db7cfa588945a1b29ec427fecfa07/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-08 20:58:13,053 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680987315735.301db7cfa588945a1b29ec427fecfa07. 2023-04-08 20:58:13,053 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 301db7cfa588945a1b29ec427fecfa07: 2023-04-08 20:58:13,053 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1680987315735.301db7cfa588945a1b29ec427fecfa07. 2023-04-08 20:58:13,223 DEBUG [RS:0;jenkins-hbase20:41091] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-08 20:58:13,423 DEBUG [RS:0;jenkins-hbase20:41091] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-08 20:58:13,455 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/meta/1588230740/.tmp/table/0d322c2f3c28438da0d9bf2fe427e5ee 2023-04-08 20:58:13,464 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0d322c2f3c28438da0d9bf2fe427e5ee 2023-04-08 20:58:13,466 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/meta/1588230740/.tmp/info/cfa55d19d8f345439bfd77bdf44d3547 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/meta/1588230740/info/cfa55d19d8f345439bfd77bdf44d3547 2023-04-08 20:58:13,473 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for cfa55d19d8f345439bfd77bdf44d3547 2023-04-08 20:58:13,473 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/meta/1588230740/info/cfa55d19d8f345439bfd77bdf44d3547, entries=22, sequenceid=93, filesize=7.1 K 2023-04-08 20:58:13,475 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/meta/1588230740/.tmp/rep_barrier/622c86e7957445aa8cabd9e01729e063 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/meta/1588230740/rep_barrier/622c86e7957445aa8cabd9e01729e063 2023-04-08 20:58:13,479 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 622c86e7957445aa8cabd9e01729e063 2023-04-08 20:58:13,479 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/meta/1588230740/rep_barrier/622c86e7957445aa8cabd9e01729e063, entries=6, sequenceid=93, filesize=5.5 K 2023-04-08 20:58:13,480 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/meta/1588230740/.tmp/table/0d322c2f3c28438da0d9bf2fe427e5ee as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/meta/1588230740/table/0d322c2f3c28438da0d9bf2fe427e5ee 2023-04-08 20:58:13,485 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0d322c2f3c28438da0d9bf2fe427e5ee 2023-04-08 20:58:13,485 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/meta/1588230740/table/0d322c2f3c28438da0d9bf2fe427e5ee, entries=9, sequenceid=93, filesize=5.5 K 2023-04-08 20:58:13,486 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.94 KB/21446, heapSize ~36.27 KB/37136, currentSize=0 B/0 for 1588230740 in 866ms, sequenceid=93, compaction requested=false 2023-04-08 20:58:13,500 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-08 20:58:13,501 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-08 20:58:13,502 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-08 20:58:13,502 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-08 20:58:13,502 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-08 20:58:13,624 INFO [RS:0;jenkins-hbase20:41091] regionserver.HRegionServer(1171): stopping server jenkins-hbase20.apache.org,41091,1680987313601; all regions closed. 2023-04-08 20:58:13,638 DEBUG [RS:0;jenkins-hbase20:41091] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/oldWALs 2023-04-08 20:58:13,638 INFO [RS:0;jenkins-hbase20:41091] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C41091%2C1680987313601.meta:.meta(num 1680987315495) 2023-04-08 20:58:13,643 DEBUG [RS:0;jenkins-hbase20:41091] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/oldWALs 2023-04-08 20:58:13,643 INFO [RS:0;jenkins-hbase20:41091] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C41091%2C1680987313601:(num 1680987314926) 2023-04-08 20:58:13,643 DEBUG [RS:0;jenkins-hbase20:41091] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:58:13,643 INFO [RS:0;jenkins-hbase20:41091] regionserver.LeaseManager(133): Closed leases 2023-04-08 20:58:13,643 INFO [RS:0;jenkins-hbase20:41091] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-04-08 20:58:13,643 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-08 20:58:13,645 INFO [RS:0;jenkins-hbase20:41091] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:41091 2023-04-08 20:58:13,650 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-08 20:58:13,650 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,41091,1680987313601 2023-04-08 20:58:13,650 ERROR [Listener at localhost.localdomain/41823-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@33411862 rejected from java.util.concurrent.ThreadPoolExecutor@377e9ad2[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2144] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-08 20:58:13,650 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-08 20:58:13,650 ERROR [Listener at localhost.localdomain/41823-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@78fdf655 rejected from java.util.concurrent.ThreadPoolExecutor@377e9ad2[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2144] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-08 20:58:13,651 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase20.apache.org,41091,1680987313601] 2023-04-08 20:58:13,651 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase20.apache.org,41091,1680987313601; numProcessing=1 2023-04-08 20:58:13,652 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase20.apache.org,41091,1680987313601 already deleted, retry=false 2023-04-08 20:58:13,652 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase20.apache.org,41091,1680987313601 expired; onlineServers=0 2023-04-08 20:58:13,652 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase20.apache.org,46121,1680987312586' ***** 2023-04-08 20:58:13,652 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-08 20:58:13,652 DEBUG [M:0;jenkins-hbase20:46121] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5cf20271, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-04-08 20:58:13,653 INFO [M:0;jenkins-hbase20:46121] regionserver.HRegionServer(1145): stopping server jenkins-hbase20.apache.org,46121,1680987312586 2023-04-08 20:58:13,653 INFO [M:0;jenkins-hbase20:46121] regionserver.HRegionServer(1171): stopping server jenkins-hbase20.apache.org,46121,1680987312586; all regions closed. 2023-04-08 20:58:13,653 DEBUG [M:0;jenkins-hbase20:46121] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 20:58:13,653 DEBUG [M:0;jenkins-hbase20:46121] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-08 20:58:13,653 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-08 20:58:13,653 DEBUG [M:0;jenkins-hbase20:46121] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-08 20:58:13,653 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1680987314625] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1680987314625,5,FailOnTimeoutGroup] 2023-04-08 20:58:13,653 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1680987314626] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1680987314626,5,FailOnTimeoutGroup] 2023-04-08 20:58:13,653 INFO [M:0;jenkins-hbase20:46121] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-08 20:58:13,653 INFO [M:0;jenkins-hbase20:46121] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-08 20:58:13,653 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-08 20:58:13,653 INFO [M:0;jenkins-hbase20:46121] hbase.ChoreService(369): Chore service for: master/jenkins-hbase20:0 had [] on shutdown 2023-04-08 20:58:13,653 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-08 20:58:13,654 DEBUG [M:0;jenkins-hbase20:46121] master.HMaster(1512): Stopping service threads 2023-04-08 20:58:13,654 INFO [M:0;jenkins-hbase20:46121] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-08 20:58:13,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-08 20:58:13,654 ERROR [M:0;jenkins-hbase20:46121] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-08 20:58:13,655 INFO [M:0;jenkins-hbase20:46121] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-08 20:58:13,655 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-08 20:58:13,655 DEBUG [M:0;jenkins-hbase20:46121] zookeeper.ZKUtil(398): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-08 20:58:13,655 WARN [M:0;jenkins-hbase20:46121] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-08 20:58:13,655 INFO [M:0;jenkins-hbase20:46121] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-08 20:58:13,655 INFO [M:0;jenkins-hbase20:46121] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-08 20:58:13,656 DEBUG [M:0;jenkins-hbase20:46121] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-08 20:58:13,656 INFO [M:0;jenkins-hbase20:46121] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-08 20:58:13,656 DEBUG [M:0;jenkins-hbase20:46121] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-08 20:58:13,656 DEBUG [M:0;jenkins-hbase20:46121] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-08 20:58:13,656 DEBUG [M:0;jenkins-hbase20:46121] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-08 20:58:13,656 INFO [M:0;jenkins-hbase20:46121] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=345.09 KB heapSize=393.84 KB 2023-04-08 20:58:13,666 INFO [M:0;jenkins-hbase20:46121] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=345.09 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/b793fe2203da4369bba924c85deadce9 2023-04-08 20:58:13,670 INFO [M:0;jenkins-hbase20:46121] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b793fe2203da4369bba924c85deadce9 2023-04-08 20:58:13,671 DEBUG [M:0;jenkins-hbase20:46121] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/b793fe2203da4369bba924c85deadce9 as hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/b793fe2203da4369bba924c85deadce9 2023-04-08 20:58:13,673 INFO [M:0;jenkins-hbase20:46121] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b793fe2203da4369bba924c85deadce9 2023-04-08 20:58:13,673 INFO [M:0;jenkins-hbase20:46121] regionserver.HStore(1080): Added hdfs://localhost.localdomain:37581/user/jenkins/test-data/cf813a00-6cf5-26fc-a2b7-95f00039838c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/b793fe2203da4369bba924c85deadce9, entries=65, sequenceid=578, filesize=24.7 K 2023-04-08 20:58:13,674 INFO [M:0;jenkins-hbase20:46121] regionserver.HRegion(2948): Finished flush of dataSize ~345.09 KB/353375, heapSize ~393.82 KB/403272, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 18ms, sequenceid=578, compaction requested=false 2023-04-08 20:58:13,675 INFO [M:0;jenkins-hbase20:46121] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-08 20:58:13,676 DEBUG [M:0;jenkins-hbase20:46121] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-08 20:58:13,678 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-08 20:58:13,678 INFO [M:0;jenkins-hbase20:46121] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-08 20:58:13,679 INFO [M:0;jenkins-hbase20:46121] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:46121 2023-04-08 20:58:13,680 DEBUG [M:0;jenkins-hbase20:46121] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase20.apache.org,46121,1680987312586 already deleted, retry=false 2023-04-08 20:58:13,752 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-08 20:58:13,752 INFO [RS:0;jenkins-hbase20:41091] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase20.apache.org,41091,1680987313601; zookeeper connection closed. 2023-04-08 20:58:13,752 ERROR [Listener at localhost.localdomain/41823-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2fd990 rejected from java.util.concurrent.ThreadPoolExecutor@377e9ad2[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2144] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-08 20:58:13,753 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): regionserver:41091-0x100925ce6e20001, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-08 20:58:13,753 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@679575c9] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@679575c9 2023-04-08 20:58:13,753 ERROR [Listener at localhost.localdomain/41823-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@50bee57 rejected from java.util.concurrent.ThreadPoolExecutor@377e9ad2[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2144] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-08 20:58:13,754 INFO [Listener at localhost.localdomain/41823] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-08 20:58:13,852 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-08 20:58:13,852 INFO [M:0;jenkins-hbase20:46121] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase20.apache.org,46121,1680987312586; zookeeper connection closed. 2023-04-08 20:58:13,852 ERROR [Listener at localhost.localdomain/41823-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1ed62238 rejected from java.util.concurrent.ThreadPoolExecutor@7ed4b9ec[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3585] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-08 20:58:13,853 DEBUG [Listener at localhost.localdomain/41823-EventThread] zookeeper.ZKWatcher(600): master:46121-0x100925ce6e20000, quorum=127.0.0.1:52440, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-08 20:58:13,853 ERROR [Listener at localhost.localdomain/41823-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@11f8b17b rejected from java.util.concurrent.ThreadPoolExecutor@7ed4b9ec[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3585] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-08 20:58:13,855 WARN [Listener at localhost.localdomain/41823] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-08 20:58:13,858 INFO [Listener at localhost.localdomain/41823] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-08 20:58:13,966 WARN [BP-1488280085-148.251.75.209-1680987310037 heartbeating to localhost.localdomain/127.0.0.1:37581] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-08 20:58:13,966 WARN [BP-1488280085-148.251.75.209-1680987310037 heartbeating to localhost.localdomain/127.0.0.1:37581] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1488280085-148.251.75.209-1680987310037 (Datanode Uuid 6ddeb571-9505-49d7-bdbc-e481d906fc0b) service to localhost.localdomain/127.0.0.1:37581 2023-04-08 20:58:13,969 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/302c12cf-64ec-a05a-dc94-df5796107026/cluster_f1ffd090-2f75-98d5-c8e5-9e16a1992906/dfs/data/data1/current/BP-1488280085-148.251.75.209-1680987310037] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-08 20:58:13,969 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/302c12cf-64ec-a05a-dc94-df5796107026/cluster_f1ffd090-2f75-98d5-c8e5-9e16a1992906/dfs/data/data2/current/BP-1488280085-148.251.75.209-1680987310037] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-08 20:58:14,006 INFO [Listener at localhost.localdomain/41823] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:0 2023-04-08 20:58:14,167 INFO [Listener at localhost.localdomain/41823] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-08 20:58:14,300 INFO [Listener at localhost.localdomain/41823] hbase.HBaseTestingUtility(1293): Minicluster is down